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ABSTRACT 


Reducing  the  logistics  burden  is  a  current  focus  for 
the  Army  as  it  works  to  develop  and  field  the  Objective 
Force.  Increasing  reliability  is  a  proven  way  to  achieve 
this  goal^  with  an  added  benefit  of  reducing  O&S  costs  and 
increasing  the  effectiveness  of  the  soldiers. 

Many  programs  have  had  difficulty  achieving  their 
required  reliability.  Operational  Testing  data  gathered  by 
the  Army  Test  and  Evaluation  Command  indicates  a  decreasing 
trend  in  achieving  reliability  requirements  with  more  than 
80%  failing  to  achieve  requirements.  It  is  intuitive  that 
it  would  be  even  more  difficult  to  achieve  ultra- 
reliability^  a  higher  level  of  reliability  and  a  proposed 
goal  of  the  Future  Combat  Systems  Program. 

To  determine  what  successful  practices  should  be  used 
to  achieve  reliability  requirements^  we  should  look  to 
successful  programs  to  show  us  the  way.  To  that  end^  this 
exploratory  study  questions  successful  Army  programs  for 
practices^  recommendations^  and  lessons  learned^  that  could 
be  shared  with  other  programs  to  achieve  reliability 
requirements . 

If  we  are  unsuccessful  in  our  endeavors  to  improve 
reliability  achievement^  future  forces  will  be 
unnecessarily  burdened  by  our  mistakes  and  incapable  of 
progress . 
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I .  INTRODUCTION 


A.  PURPOSE 

This  research  focuses  on  reliability  aspects  of  the 
Future  Combat  Systems  (FCS)  and  analyzes  the  strategies 
that  were  used  to  achieve  it.  This  research  is  applicable 
to  other  weapon  system  programs^  especially  those  where  the 
goal  is  to  reduce  the  logistics  burden  by  developing  and 
fielding  a  reliable  system. 

B .  BACKGOUND 

The  U.S.  Army^  as  well  as  the  other  services^  has  been 
plagued  with  a  large  logistics  burden.  The  principle 
logistics  burdens  are  the  fuel^  ammunition^  food^  water^ 
and  spare  parts  necessary  to  sustain  an  operational 
military  force.  Because  of  uncertainties  in  combat^  plans 
for  stockpiles  of  supplies  are  designed  to  meet  any 
contingency.  As  a  result^  large  amounts  of  supplies  and 
logistics  personnel  are  moved  into  theater  to  support  the 
combat  troops  and  equipment  on  a  just-in-case  basis.  [Ref. 
l:p.  26] 

It  is  anticipated^  future  forces  will  be  called  upon 
to  be  inserted  into  the  combat  zone  by  air^  and  must  be 
prepared  to  accomplish  a  numerous  variety  of  combat 
missions^  operating  for  up  to  14-days  without  logistics 
support  from  outside  the  battle  area.  Because  of  this 
requirement^  future  forces  and  systems  must  require  less 
logistical  support  than  their  predecessors.  [Ref.  l:p.  27] 

Reliability  is  considered  a  key  focus  for  the 
reduction  of  the  logistics  burden.  Reliability  is  defined 
as  the 
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probability  that  an  item  will  perform  its 
intended  function  for  a  specified  interval  under 
stated  conditions.  [Ref.  2:p.  36] 

With  more  reliable  systems,  less  spares  and  maintenance 
assets  would  be  required  for  support,  thereby  reducing  the 
logistics  burden. 

The  Army  is  focused  on  reducing  the  logistics  burden 
of  currently  fielded  systems  and  future  systems  under 
development.  One  of  these,  the  Future  Combat  Systems  (FCS) 
is  a  weapon  system  currently  under  development  as  part  of 
the  Army's  Objective  Force.  The  FCS  may  consist  of  as  many 
as  five  separate  robotic  systems  that  interface  together 
through  a  central  manned  command  and  control  vehicle. 
Because  of  its  autonomous  function,  the  FCS  must  achieve 
high  reliability.  To  this  end,  the  Army  Science  Board 
recommended  in  a  study  that  the  FCS  establish  ultra¬ 
reliability  as  a  Key  Performance  Parameter  (KPP) . 

Many  programs  have  had  difficulty  achieving  their 
required  reliability.  It  is  intuitive  that  it  would  be 
even  more  difficult  to  achieve  ultra-reliability,  a  higher 
level  of  reliability.  Operational  Testing  data  gathered  by 
the  Army  Test  and  Evaluation  Command  depicts  that  during 
the  period  of  1985  to  1990  41%  of  systems  met  reliability 
requirements.  This  figure  worsened  to  20%  meeting 
reliability  requirements  during  the  period  of  1996  to  2000. 
Worse  yet,  those  that  failed  to  meet  reliability 
requirements  also  failed  to  meet  even  50%  of  the 
reliability  goals.  [Ref.  3] 

Because  many  programs  were  unable  to  achieve 
reliability  requirements,  LTG  Kern,  the  Military  Deputy  to 
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the  Assistant  Secretary  of  the  Army  for  Acquisition, 
Logistics,  and  Technology,  initiated  a  study  to  determine 
what  should  be  done  to  remedy  the  situation. 

C .  RESEARCH  QUESTIONS 

1 .  Primary  Research  Question 

What  strategies  should  be  used  to  achieve  reliability 
requirements  for  weapon  system  development? 

2 .  Secondary  Research  Questions 

a.  What  is  reliability  and  how  does  it  affect 
total  life  cycle  costs  and  the  logistics  burden? 

b.  Why  is  ultra-reliability  a  recommended  goal 
for  the  FCS  program? 

c.  Should  ultra-reliability  be  a  goal  of  weapon 
system  programs? 

d.  What  is  the  guidance  for  achieving 
reliability  requirements? 

e.  How  have  successful  weapon  system  programs 
achieved  reliability  requirements? 

i.  How  did  programs  organize  to  achieve 
reliability  requirements? 

ii .  How  were  reliability  requirements 

developed? 

iii.  What  management  plans  for  achieving 
reliability  requirements  were  developed  and  implemented? 

iv.  What  processes  were  used  to  achieve 
reliability  requirements  and  how  was  this  measured? 
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V.  Where  in  the  Acquisition  Process  was 
the  program  focused  on  achievement  of  reliability 
requirements? 

f.  What  are  the  best  strategies  that  should  be 
used  for  the  FCS  program  as  well  as  other  weapon  system 
programs? 

D .  SCOPE 

The  scope  of  the  study  will  include:  (1)  an  in-depth 

analysis  of  the  logistics  burden  and  the  effect  of 
reliability  upon  it;  (2)  identification  and  analysis  of  the 
acquisition  lifecycle  for  reliability  inputs;  (3) 
identification  and  interviews  of  successful  programs  to 
identify  their  focus  on  reliability;  (4)  review  of 
advantages  and  disadvantages  of  the  approaches  used  by  the 
programs.  The  thesis  will  conclude  with  a  recommendation 
for  strategies  focusing  on  improving  reliability  for  the 
Future  Combat  Systems  and  other  weapon  system  programs . 

E .  METHODOLOGY 

The  methodology  utilized  in  this  thesis  research  is 
the  following: 

1.  Conduct  a  literature  search  of  books,  magazine 
articles,  CD-ROM  systems,  and  other  library  information. 

2.  Conduct  an  Internet  search  of  data  pertaining  to 
the  Future  Combat  Systems,  reliability,  and  reliability 
guidance . 

3.  Identify  the  reliability  requirements  of  the  Future 
Combat  Systems . 
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4.  Utilize  Director  of  Combat  Development  (DCD) ^ 
Training  and  Doctrine  Command  (TRADOC) ,  and  Army  Test  and 
Evaluation  Command  (ATEC)  reliability  experts. 

5.  Utilize  ATEC  Operational  Test  Data  to  identify 
programs  that  successfully  met  reliability  requirements. 

6.  Develop  interview  questions  to  obtain  data  to 
answer  the  primary  and  secondary  research  questions. 

7.  Question  those  successful  programs  for  strategies 
for  the  achievement  of  reliability  requirements. 

8.  Evaluate  the  methodologies  by  analyzing  the  data. 

9.  Compare  responses  to  literature  review  and  one 
another . 

10.  Formulate  recommendations  based  on  the  analysis  of 
the  thesis. 

F.  ORGANIZATION 

Chapter  I.  Introduction.  This  chapter  presents  the 
problem  of  the  logistics  burden  and  the  reliability 
component  impact.  It  also  outlines  the  ECS  and  its  goal  of 
reducing  the  logistics  burden.  The  remainder  of  the  thesis 
organization  is  outlined  below. 

Chapter  II.  Reliability  Its  Effects^  and  Guidance 
for  the  Achievement  of  Requirements .  This  chapter  provides 
the  background  of  reliability  by  defining  and  describing 
reliability  and  its  effects  on  the  logistics  burden  and 
lifecycle  costs.  It  also  outlines  both  mandatory  and 
discretionary  guidance  for  the  achievement  of  reliability 
requirements . 
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Chapter  III.  Methodology.  This  chapter  outlines  the 
methodology  of  the  interviews.  It  identifies  how  the 
interviews  occurred^  the  interview  subjects^  and  the 
questions  posed. 

Chapter  IV.  Data  Summary .  In  this  chapter^  the 
subject's  responses  to  the  interview  questions  are 
summarized . 

Chapter  V.  Data  Analysis.  Data^  in  the  form  of 
interview  responses^  is  analyzed  to  discover  trends  which 
are  then  compared  to  published  guidance.  Responses  are 
also  analyzed  to  identify  advantages  and  disadvantages. 

Chapter  VI.  Conclusions  and  Recommendations.  This 
chapter  restates  the  primary  and  secondary  research 
questions^  summarizes  the  findings^  and  makes 
recommendations  for  implementation . 

G.  BENEFITS 

The  achievement  of  reliability  requirements  is  a 
problem  that  is  currently  being  studied  by  the  Army  due  to 
the  inability  of  many  programs  to  achieve  reliability 
requirements.  This  study  will  review  the  importance  of 
reliability  and  its  effect  on  the  logistics  burden  and  a 
program's  life  cycle  costs.  It  will  also  examine  what 
guidance  exists  and  what  strategies  successful  programs  are 
using  to  achieve  reliability  requirements.  The  study  will 
conclude  with  recommendations  of  successful  strategies  to 
achieve  reliability  requirements  for  use  by  the  FCS  and  all 
weapon  system  programs. 
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II.  RELIABILITY,  ITS  EFFECTS,  AND  GUIDANCE  FOR  THE 
ACHIEVEMENT  OF  REQUIREMENTS 


A.  INTRODUCTION 

The  Army,  in  attempting  to  field  its  Objective  Force  - 
the  force  of  the  future  -  has  focused  its  efforts  on 
reducing  the  logistics  burden.  One  prescribed  method  for 
achieving  this  is  to  make  systems  under  development  more 
reliable.  Systems  that  are  reliable  have  fewer  failures, 
require  less  maintenance  and  spares,  and  have  greater 
availability.  Because  of  this,  reliability  can  act  as  a 
combat  multiplier  and  also  have  a  profound  effect  on  the 
reduction  of  the  logistics  burden  and  life  cycle  costs. 

Few  acquisition  programs  have  achieved  reliability 
requirements  in  the  last  decade.  Failure  to  achieve  these 
requirements  can  result  in  the  fielding  of  a  system  that  is 
not  operationally  suitable,  delay  of  fielding  to  correct 
deficiencies,  or  cancellation  of  the  program. 

The  Future  Combat  Systems  (FCS)  Program,  in  the  early 
stages  of  development,  will  attempt  to  develop  a  highly 
reliable  combat  system.  The  goal  is  to  field  a  system  with 
a  reduced  logistics  burden,  benefiting  the  Army  by 
requiring  less  logistics  support  than  currently  fielded 
systems . 

This  chapter  provides  an  overview  of  reliability  and 
its  effects  on  the  logistics  burden  and  lifecycle  costs, 
introduces  the  FCS  and  its  reliability  requirements,  and 
outlines  reliability  guidance  and  where  reliability  is 
encountered  in  the  acquisition  life  cycle. 
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B.  RELIABILTY  AND  ITS  EFFECTS  ON  THE  LOGISTICS  BURDEN  AND 

LIFE  CYCLE  COSTS 

1 .  Introduction  to  Reliability 

The  most  widely  used  definition  of  reliability  is  the 

probability  that  an  item  will  perform  its 

intended  function  for  a  specified  interval  under 

stated  conditions.  [Ref.  2:p.  36] 

The  military  once  had  its  own  definition  of 
reliability,  which  it  defined  in  Military  Standard  (MIL- 
STD)  721-C,  Definitions  of  Terms  for  Reliability  and 
Maintainability,  but  chose  to  cancel  it. 

In  weapon  system  documents,  the  military  also  uses 
terms  such  as  mission  reliability  and  a  new  concept,  ultra¬ 
reliability.  Mission  reliability  is  often  defined  as  "the 
probability  that  a  system  will  perform  mission-essential 
functions  for  a  period  of  time  under  conditions  stated  in 
the  mission  profile."  [Ref.  4:  p.  10-3]  It  is  commonly 
accepted  that  ultra-reliability  is  an  exceptionally  high 
reliability,  with  an  extremely  large  Mean  Time  Between 
Failure.  The  National  Aeronautic  Space  Administration 

(NASA)  refers  to  ultra-reliability  in  software  as  having  a 
failure  probability  during  a  one-hour  test  mission  of  less 
than  0.0000001.  This  means  that  the  software  will  not  fail 
99.99999%  of  the  time.  [Ref.  5] 

Reliability  probabilities  for  various  components  can 
be  modeled  using  different  reliability  distributions, 
depending  upon  their  characteristics.  The  two  most  common 
are  the  exponential  distribution  and  the  Weibull 
distribution.  The  exponential  distribution,  the  simplest 
to  model,  describes  reliability  failures  in  systems  such  as 


for  electronics,  which  exhibit  a  constant  failure  rate. 
The  exponential  reliability  distribution  is  expressed  as: 

R(t)  = 

where  R  is  the  reliability,  t  is  time,  and  X  is  1/Mean 
Time  Between  Failure  (MTBF)  .  [Ref.  2:p.  39]  MTBF  is  the 
mean  time  that  a  system  will  successfully  perform  its 
function  before  failing  and  is  usually  expressed  in  hours. 
Reliability  is  therefore  an  exponential  distribution  that 
is  a  function  of  MTBF  and  time,  and  as  time  increases, 
reliability  decreases.  It  can  therefore  be  seen  that 
MTBF  is  a  key  parameter  of  reliability. 

If  in  the  above  equation  the  MTBF  was  1000  hours,  X 
would  be  1/1000  or  0.001  and  the  resulting  reliability 
after  100  hours  is  predicted  by  the  exponential  reliability 
distribution  to  be  about  0.90.  This  means  that  after  100 
hours  of  operation,  90%  of  the  time  the  item  will  still  be 
operational.  After  1000  hours,  the  reliability  would  be 
about  0.36,  or  36%  of  the  time  it  will  be  operational. 

System  reliability  is  dependent  upon  the  reliability 
of  the  system' s  subcomponents  and  the  way  they  are 
connected.  There  are  two  primary  ways  of  connecting 
subcomponents,  these  are  in  series  and  parallel.  Series 
connection  is  most  common,  with  parallel  connection  of 
critical  components  to  provide  redundancy  and  therefore, 
increased  reliability.  [Ref.  2:p.  41]  In  the  figure  below, 
even  though  the  subcomponents  in  series  connection  have  a 
higher  reliability  than  the  two  connected  in  parallel,  the 
overall  system  reliability  for  those  in  series  is  less  than 
for  those  in  parallel. 
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Figure  1.  Series  vs.  Parallel  Design 


To  determine  the  reliability  of  a  system,  the 
individual  reliability  of  the  subcomponents  in  series  are 
multiplied  together,  while  for  those  in  parallel,  the 
subcomponent  unreliabilities  (1-R)  are  multiplied  together 
and  the  unreliability  is  taken  again  (l-(l-R)),  resulting 
in  the  predicted  reliability.  [Ref.  6:p.  32]  The 

reliability  of  a  system  with  subcomponents  in  series  can  be 
no  greater  than  the  reliability  of  the  least  reliable 
component.  The  reliability  of  a  system  can  be  improved  by 
placing  components  in  parallel.  This  increases  the 

reliability  because  if  one  component  fails,  the  other 
component,  as  well  as  the  system,  will  continue  operating. 

The  design  of  equipment  and  weapon  systems  has 
increased  in  complexity  over  time,  which  may  also  reduce 
reliability.  Take  for  example  table  one,  a  tractor  that 
only  had  1,200  critical  parts  in  1935  now  may  have  more 
than  2,900  critical  parts.  This  reduces  the  tractor's 
reliability  from  88.7%  in  1935  to  74.8%,  assuming  an 
average  component  reliability  of  99.99%  and  critical 
components  arrayed  in  series.  Worse  yet,  this  translates 
to  more  than  twice  the  number  of  tractors  failing  per  1,000 
tractors  now  as  compared  to  1935.  A  similar  prediction  may 
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be  made  when  comparing  military  weapon  systems  such  as  the 


M4A1  Sherman  tank  with  the  M1A2  Abrams  tank. 


Farm 

tractor 

model 

year 

Number  of 

critical 

components 

Tractor  reliability 
per  year, *  assuming 
an  average  component 
reliability  of 

99.99% 

No.  of  tractors 
failing  per  year 
per  1,000 
tractors 

1935 

1,200 

88.7% 

113 

1960 

2,250 

79.9% 

201 

1970 

2,400 

78 . 7% 

213 

1980 

2,600 

77 . 1% 

229 

1990 

2,900 

74 . 8% 

252 

is  assumed  that  all  critical  components  are 
reliabilitywise  in  series. 

Table  1.  Tractor  Design  Complexity  [From  Ref.  7:p.  11] 


To  illustrate  the  effect  of  critical  subcomponent 
reliability  on  the  system,  consider  Table  2.  If  1,000 
critical  subcomponents,  each  having  a  reliability  of 
99.999%,  are  arrayed  in  series,  the  system  reliability  is 
99.01%.  Now  if  there  are  100,000  critical  subcomponents, 
with  the  same  reliability,  arrayed  in  series,  the  system 
reliability  is  reduced  to  only  36.79%.  This  figure  is  even 


worse  if  the  subcomponent  reliability  is  only  99.0%. 


Number  of 

critical 

components 

Individual  component  reliability 

99.999% 

99.99% 

99.9% 

99.0% 

System  reliability 

10 

99.99% 

99.90% 

99.00% 

90 . 44% 

100 

99.90% 

99.01% 

90.48% 

36.60% 

250 

99.75% 

97.53% 

77 . 87% 

8 .11% 

500 

99.50% 

95.12% 

60 . 64% 

0 . 66% 

1,000 

99.01% 

90.48% 

36.77% 

<0.1% 

10,000 

90.48% 

36.79% 

<0.1% 

<0.1% 

100,000 

36.79% 

<0.1% 

<0 . 1% 

<0 . 1% 

*It  is  assumed  that  all  critical  components  are 
reliabilitywise  in  series. 


Table  2.  Component  Reliability  and  Design  Complexity  [From 

Ref.  7 : p .  12] 

2 .  The  Effect  of  Reliability  on  the  Logistics  Burden 
and  Lifecycle  Costs 


11 


As  the  figure  below  indicates.  Operation  and  Support 
(O&S)  costs  can  be  72%  of  the  life  cycle  cost  (LOG)  of  a 
system.  The  diagram  is  somewhat  dated,  using  data  from 
1980,  however,  the  figures  still  remain  accurate  for  most 
programs . 


Life  Cycle  Cost 


OPERATION  AND  SUPPORT 


SYSTEM  ACQUISITION 


I 


72% 


I 


28% 


1 — i  i  i 

0  . 

MILESTONES 


SOURCE:  John  F.  Phillips,  DUSD  (L)  9/96 


DISPOSAL 


YEARS 


Figure  2.  Nominal  Cost  Distribution  (Typical  1980  DoD 

Acquisition  Program  with  a  service  life  of  about  30  years) 

[From  Ref.  7:p.  13-6] 

O&S  costs  are  the  resources  necessary  for  the 
operation  and  support  of  the  system,  subsystem,  or  major 
component  during  its  useful  life.  These  resources  may 
consist  of  such  things  as  fuel,  spare  parts  and 
maintenance,  among  others. 


Managing  and  reducing  O&S  costs  is  a  focus  of  current 
weapon  development  programs .  Buying  into  a  weapon  system 
may  be  relatively  inexpensive,  but  operating  it  and 
maintaining  it  can  be  burdensome.  Examining  the  Army 
budget  supports  this  statement. 
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O&S  costs  are  a  large  part  of  the  Army  budget, 
accounting  for  almost  33%  of  the  entire  budget.  By 
reducing  O&S  costs,  the  Army  would  be  able  to  spend  money 
on  more  critical  needs,  such  as  force  modernization  or 
quality  of  life.  O&S  costs  are  also  rising  partially  as  a 
result  of  less  reliable  equipment.  For  fiscal  year  2002, 
the  Army  requested  $26. 7B  for  Operation  and  Maintenance 
(also  known  as  O&S)  ,  an  increase  of  almost  12%  from  the 


previous  year. 


FYOl  vs.  FY02  TOTAL  OBLIGATION  AUTHORITY 
($  billions) 

APPROPRIATION 

FYOl 

FY02 

Military  Personnel 

$28.4 

$30.2 

Operation  and  Maintenance 

23.9 

26.7 

Procurement 

11.0 

11.2 

Research,  Development,  Test  and 
Evaluation  (RDT&E) 

6.3 

6.7 

Military  Construction 

1.3 

2.1 

Army  Family  Housing 

1.2 

1.4 

Chemical  Demilitarization 

1.0 

1.2 

Base  Realignment  and  Closure 

.3 

.2 

Environmental  Restoration 

.4 

.4 

Total  * 

$73.7 

$80.2 

Table  3.  Army  FYOl  vs.  FY02  Total  Obligation  Authority 

[From  Ref.  8] 

Initial  investments  in  the  design  and  production  of 
reliable  weapon  systems  can  have  an  enormous  result  by 
reducing  O&S  costs.  Examples  of  this  are  found  in  the 
Minuteman  I  missile  system,  which  estimated  that  for  every 
dollar  invested  in  the  reliability  improvement  program 
resulted  in  a  return  of  eight  dollars.  Over  a  ten-year 
period,  the  net  savings  was  expected  to  be  $160,000,000. 
Another  example  of  this  savings  is  that  of  the  Atlas 
Guidance  System.  The  result  of  an  annual  investment  of 
$10,100,000  in  a  "high"  reliability  program  during 
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development  and  production  resulted  in  an  annual  savings  of 

$58,400,000  after  fielding.  A  last  example  is  of  the  F-105 

weapon  system,  which,  by  implementing  a  reliability 

improvement  program,  increased  the  reliability  of  the 

system  from  0.7263  to  0.8986,  resulting  in  an  annual 

savings  estimated  at  $25,500,000.  [Ref.  7:pp.  23-24] 

C.  THE  FUTURE  COMBAT  SYSTEMS  AND  THE  NEED  FOR  ULTRA¬ 
RELIABILITY 

1.  The  Future  Combat  Systems  (FCS)  Program 

The  FCS  Program  is  a  cooperative  development  program 
between  the  Defense  Advanced  Research  Projects  Agency 
(DARPA) ,  and  the  United  States  Army.  DARPA,  through  its 
Objective  Force  Program  Office  (PM-OF)  ,  is  responsible  for 
the  management  of  the  first  phase  of  the  system's 
lifecycle,  the  Concept  and  Technology  Development  Phase. 

FCS  AcQidsition  C€mc^jt 


Figure  3.  FCS  Program  Schedule  [From  Ref.  9] 

The  Army,  through  the  Future  Combat  Systems  Program  Office 
(PM-FCS),  is  developing  a  streamlined  acquisition 
management  approach  for  the  program.  [Ref.  9] 
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The  purpose  of  the  program  is  to  develop  a  new, 
innovative,  advanced  warfighting  capability  by 
incorporating  leap-ahead  technologies  focused  on  unmanned 
systems.  The  program  is  a  Simulation  Based  Acquisition 
(SBA) ,  whereby  contractors  and  the  Government  will  utilize 
modeling  and  simulation  (M&S)  to  design  and  develop  the 
force  and  reduce  risk. 

FCS  is  defined  as  a 

networked  systems  of  systems  that  will  serve  as  a 
core  building  block  within  all  maneuver  Unit  of 
Action  echelons  to  develop  overmatching  combat 
power,  sustainability,  agility,  and  versatility 
necessary  for  full  spectrum  military  operations. 

[Ref.  9] 

FCS  is  envisioned  as  an  ensemble  of  manned  and 
unmanned  systems  that  will  fulfill  the  ground  component 
requirement  of  the  Army's  Objective  Force.  Key  tenets  of 
the  FCS  are  "survivability,  lethality,  deployability, 
agility,  sustainability,  versatility,  and  responsiveness." 
[Ref.  9] 

FCS  will  be  comprised  of  "networked  space-,  air-  and 
ground-based  maneuver,  maneuver  support  and  sustainment 
systems."  [Ref.  9] 

A  commonly  presented  concept  is  depicted  in  the  figure 
below.  It  incorporates  both  an  unmanned  aerial  vehicle 
(UAV)  and  a  ground  based  robotic  vehicle  that  provide 
reconnaissance,  surveillance,  and  target  acquisition 
(RSTA) .  A  robotic  vehicle  equipped  with  missiles  provides  a 
beyond  line-of-sight  (BLOS)  indirect  fire  capability  while 
another  robotic  vehicle  equipped  with  a  main  gun  provides  s 
line-of-sight  (LOS)  direct  fire  capability.  A  central 


15 


command  and  control  vehicle  transports  the  dismount 
soldiers  and  provides  command,  control,  communications 
interface  (C3I)  for  the  entire  system. 


Figure  4.  FCS  Commonly  Presented  Concept  [From  Ref.  10] 

2 .  The  need  for  Ultra-reliability 

Because  FCS  relies  on  leap-ahead  technologies  and 
unmanned,  robotic  vehicles,  reliability  is  a  chief  concern 
of  the  system.  Reliability  is  also  necessary  to  ensure  a 
reduced  logistics  burden  and  acceptable  lifecycle  costs. 
For  these  reasons,  the  Hon.  Paul  J.  Hoeper,  Assistant 
Secretary  of  the  Army  (Acquisition,  Logistics  and 
Technology)  (ASA (ALT))  requested  that  the  Army  Science 
Board  convene  a  Year  2000  Summer  Study  to  study  "Technical 
and  Tactical  Opportunities  for  Revolutionary  Advances  in 
Rapidly  Deployable  Joint  Ground  Forces  in  the  2015-2025 
Era."  [Ref.  11]  One  focus  that  the  ASA  (ALT)  requested  the 
board  examine  was  the  Sustainment  and  Support  of  the  future 
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forces.  The  goal  of  this  investigation  was  to  identify  a 
sustainment  and  support  capability  with  a  reduced  logistics 
burden,  a  means  to  provide  future  forces  with  a... 

significantly  greater  systems  reliability  ...  along 
with  graceful  degradation  and  ultra-reliability 
leading  to  simplified  battlefield  maintenance, 
repair,  and  diagnostics/prognostics  (including 
disposable  /  expendable  components  /  systems) . 

[Ref.  11] 

The  Army  Science  Board  focused  on  three  problems  in 
dealing  with  FCS  Sustainment  and  Support  issues,  one  of 
which  was  the  reduction  in  demand  for  materiel  and  support 
of  the  FCS.  A  proposed  solution  was  that  the  FCS  be 
designed  for  supportability  with  a  built-in  or  inherent 
reliability  allowing  operation  for  seven  days  or  more 
without  maintenance  and  support.  Ultimately,  the  board 
recommended  that  the  FCS  achieve  ultra-reliability  to 
reduce  the  logistics  burden.  The  reason  for  this 
recommendation  was  because  ultra-reliability  reduces 
maintenance  and  support  requirements.  It  reduces  the 
number  of  maintenance  personnel,  equipment,  and  spares 
required  for  support.  Some  estimates  indicate  that  ultra¬ 
reliability  can  reduce  service  and  support  personnel 
requirements  in  the  Objective  Force  Area  of  Operations  by 
as  much  as  83%.  [Ref.  12:p.  32] 

The  Army  Science  Board  further  recommended  four 
strategies  for  achieving  ultra-reliability. 
Tradeoff /payoff  analysis  using  models  and  simulations  is 
one  recommended  method  for  realizing  that  the  benefits  of 
achieving  ultra-reliability  exceed  the  costs.  A  second 
strategy  is  the  use  of  prognostics  and  diagnostics,  through 
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the  use  of  imbedded  sensors.  This  will  allow  maintenance 
actions  to  occur  prior  to  part  failure  due  to  failure 
prediction  and  analysis.  A  third  strategy  is  the 
application  of  science  and  engineering  principles  during 
system  design  and  development  to  achieve  inherent 
reliability  requirements.  The  last  recommended  strategy  is 
the  specification  of  reliability  as  a  Key  Performance 
Parameter  (KPP)  during  procurement  and  acquisition.  [Ref. 
12:pp.  42-45] 

Ultimately,  the  board  concluded  that  ultra-reliability 
should  be  a  KPP  for  the  FCS .  However,  many  senior  Defense 
and  Army  logisticians,  to  include  the  Office  of  the  Deputy 
Chief  of  Staff  (Operations)  (ODCSOPS) ,  Office  of  the  Deputy 
Chief  of  Staff  (Procurement)  (ODCSPRO) ,  ASA(ALT),  and  the 
Objective  Force  Task  Force  disagreed  with  this 
recommendation.  The  reasons  provided  for  their 
disagreement  are  that  it  is  too  early  in  the  lifecycle  to 
designate  a  KPP,  it  reduces  the  trade  space,  and  it 
constrains  the  Program  Manager's  flexibility.  Only  the 
Headquarters,  Army  Materiel  Command/Army  Materiel  Systems 
Analysis  Activity  (HQ  AMC/AMSAA)  and  the  Office  of  the 
Deputy  Chief  of  Staff  (Logistics)  (ODCSLOG)  agreed  that 
ultra-reliability  should  be  a  KPP.  [Ref.  3] 

3 .  FCS  Reliability  Requirements 

The  DARPA/Army  FCS  Program  Solicitation,  released  in 
November  2001,  emphasized  the  need  for  reliable  systems  in 
the  development  of  the  FCS.  It  recognized  that  FCS 
reliability  was  "critical  to  reducing  [the]  logistics  foot 
print  and  lifecycle  cost."  [Ref.  9:p.  76] 
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To  achieve  this,  the  solicitation  required  the  FCS 
have  the  following  reliability  capabilities: 

1.  ultra-reliable  and/or  redundant  components  to 
remain  operationally  effective  for  the  full  3/7- 
day  mission  period  with  minimal  pulsed  service  or 
repair  organic  to  the  Unit  of  Action.  [Ref.  9:p. 

76] 

2.  reduce  demand  and  minimize  the  maneuver 
sustainment  burden  on  unit  effectiveness  through 
balanced  system  reliability,  redundancy  and 
repair,  to  include  embedded  diagnostics  and 
prognostics  as  well  as  modular  component  design. 

[Ref.  9 : p .  76] 

Specified  reliability  deliverables  required  the 
contractor  to: 

1.  Propose  reliability,  maintainability  and 
testability  criteria,  e.g.,  confidence  levels  and 
no  evidence  of  failure.  [Ref.  9:p.  76] 

2.  Identify  recommended  reliability, 
maintainability  and  testability  criteria  to  be 
tested  in  SDD  prototype  testing  (PQT  and  LUT)  . 

[Ref.  9:p.  76] 

D .  MANDATORY  AND  DESCRETIONARY  GUIDANCE  FOR  ACHIEVING 
RELIABILITY  REQUIREMENTS 

There  are  many  sources  of  reliability  guidance  for  the 
military  to  assist  programs  in  achieving  reliability 
requirements.  The  amount  of  mandatory  guidance,  however, 
has  decreased  as  a  result  of  acquisition  reform. 

This  section  will  present  the  sources  of  guidance  and 
the  nature  of  the  guidance,  whether  it  is  mandatory  or 
discretionary.  It  will  also  identify  where  in  the 
lifecycle  of  a  program  the  Program  Manager  should  implement 
the  guidance . 
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1 .  Mandatory  Guidance 


There  are  four  mandatory  sources  of  reliability 
guidance  relevant  to  the  Army.  Two  are  Department  of 
Defense  Directives^  DoDD  5000.1^  The  Defense  Acquisition 
System^  and  DoD  5000. 2-R^  Mandatory  Procedures  for  Major 
Defense  Acquisition  Programs  (MDAPs)  and  Major  Automated 
Information  System  (MATS)  .  Two  are  mandatory  Army 
Regulations  are  AR  70-1^  Army  Acquisition  Policy  and  AR  71- 
9r  Materiel  Requirements .  [Ref.  13] 

DoDD  5000.1^  directs  that 

acquisition  program  managers  shall  focus  on 
logistics  considerations  early  in  the  design 
process  to  ensure  that  they  deliver  reliable 
systems  that  can  be  cost-effectively  supported 
and  provide  users  with  the  necessary  support 
infrastructure  to  meet  peacetime  and  wartime 
readiness  requirements.  [Ref.  14] 

DoD  5000. 2-R,  directs  the  PM  to 

develop  RAM  system  requirements  based  on  the 
Operational  Requirements  Document  (ORD)  and  TOC 
(Total  Ownership  Cost)  consideration^  and  state 
them  in  quantifiable^  operational  terms^ 
measurable  during  development  and  operational 
T&E.  [Ref.  15] 

it  further  states  ''reliability  requirements  shall  address 
mission  reliability  and  logistic  reliability."  [Ref.  15] 

AR  70-1  states 

when  reliability  and  maintainability  requirements 
are  included  in  solicitations^  they  should  be 
included  by  specifying: 

quantified  reliability  and  maintainability 
requirements  and  allowable  uncertainties  (such  as 
statistical  risks) ^ 
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failure  definitions  and  thresholds  (FDSC) ,  and 


life-cycle  usage  conditions  (OMS/MP) .  [Ref.  16:p. 

18] 

It  further  states  that 

solicitations  should  not  cite  any  specification, 
standard  or  handbook  or  include  language 
specifying  'how  to'  design,  manufacture  or  test 
for  reliability.  MIL  HBK  217,  Reliability 

Prediction  of  Electronic  Equipment ,  is  not  to 
appear  in  any  solicitation  as  it  has  been  shown 
to  be  unreliable  and  its  use  can  lead  to 
erroneous  and  misleading  reliability  predictions. 

[Ref.  16:p.  19] 

AR  70-1,  states  in  paragraph  5-13  Performance-based 
requirements,  "where  certain  critical  processes  must  be 
contractually  required  in  order  to  protect  both  parties' 
interest,"  [Ref.  16:p.  20]  in  considering  program 
complexity  and  risk,  offerors  may  be  required  to  first  use 
their  own  processes,  secondly  use  industry  accepted 
standards,  and  lastly  use  Government  developed  processes  to 
achieve  key  attributes  or  performance  parameters.  Because 
the  objective  is  to  encourage  the  use  of  non-Governmental 
processes,  the  program  manager  must  obtain  a  waiver  from 
the  Milestone  Decision  Authority. 

In  comparison  to  section  5-8,  Defining  R&M 
Requirements,  this  latitude  is  not  given  for  achieving  R&M 
requirements.  Contractors  are  instead  expected  to  achieve 
the  requirements  on  their  own. 

AR  71-9  directs  that 

an  effective  R&M  program  that  focuses  on 
achievement  of  operational  requirements  and  O&S 
cost  targets  is  necessary  to  ensure  that  user 
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operational  reliability  requirements  will  be  met. 
CBTDEVs  and  TNGDEVs  will  participate  with  MATDEVs 
in  defining  an  effective,  tailored  R&M  program 
for  each  system  pursued.  [Ref.  17 :p.  14] 

The  regulation  also  goes  on  to  state 

CBTDEV/TNGDEV  will  provide  an  operational  mode 
summary/mission  profile  (OMS/MP)  and  a  failure 
definition  and  scoring  criteria  (FDSC)  to  support 
the  reliability  requirement.  [Ref.  17 :p.  14] 

2 .  Discretionary  Guidance 

In  addition  to  the  mandatory  guidance,  there  are  many 
discretionary  sources  of  reliability  guidance.  These 
discretionary  sources  consist  mostly  of  Department  of  the 
Army  Pamphlets  and  Military  Handbooks.  Because  there  are 
so  many,  only  some  of  the  most  relevant  sources  will  be 
identified. 

DA-PAM  70-3,  Army  Acquisition  Procedures,  devotes  an 
entire  section  to  Reliability,  Maintainability  and 
Availability.  It  provides  discretionary  guidance 
concerning  establishment  of  Reliability  and  Maintainability 
(R&M)  requirements,  management,  engineering  and  design, 
testing,  integrated  process  teams  and  their  assessment. 
[Ref.  18:pp.  99-105] 

There  are  also  several  military  handbooks  that  address 
reliability.  These  include  MIL-HDBK-781A,  Handbook  for 
Reliability  Test  Methods ,  Plans,  and  Environments  for 
Engineering  Development ,  Qualification ,  and  Production ,  and 
MIL-HDBK-189,  Reliability  Growth  Management. 

MIL-HDBK-781A  provides  a  list  of  reliability  test 
methods,  reliability  test  plans,  and  environmental  profile 
data  to  use  as  a  guide  when  testing  systems  for  contractual 
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reliability  requirements  during  developmental  testing.  The 
reliability  test  methods  and  plans  include  such  things  as 
growth  monitoring  and  MTBF  assurance  testing.  Test 
environmental  profiles  are  provided  for  test  environments 
for  equipment^  vehicles^  and  aircraft.  This  testing  is 
necessary  to  identify  defects  and  failures  so  they  may  be 
corrected  prior  to  fielding.  The  information  contained  in 
the  handbook  is  applicable  to  all  systems  and  can  be 
tailored  for  any  program.  [Ref.  19] 

MiL-HDBK-189  outlines  reliability  growth  concepts  and 
methodologies  for  management  of  reliability  growth  during 
the  developmental  stage.  it  first  presents  the  fundamental 
concepts  followed  by  details  for  concept  implementation. 
[Ref.  20] 

There  is  not  as  much  regulatory  guidance  as  there  has 
been  in  the  past.  Many  mandatory  standards  and 
specifications  have  been  canceled  or  made  discretionary  as 
a  result  of  acquisition  reform. 

3.  Acquisition  Reform  and  Its  Effects 

Acquisition  Reform  has  had  a  great  impact  on  the 
management  of  acquisition  programs.  Key  tenants  of 
Acquisition  Reform  are  ''better^  faster^  cheaper."  With 
these  objectives^  many  changes  were  made  to  the  acquisition 
process  in  an  attempt  to  improve  the  process. 

One  significant  change  was  the  dependence  on 
commercial  industry  to  use  their  practices  and  standards  in 
favor  of  military  standards  and  specifications.  This 
change  was  made  after  commercial  industry  identified 
military  detailed  specifications  as  unnecessary  cost 
drivers.  in  September  1985^  the  Under  Secretary  of  Defense 
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(Acquisition,  Logistics  &  Technology) ,  mandated  that  the 
preference  for  specifications  would  be  first  for 
performance  specifications,  then  commercial  specifications, 
then,  as  a  last  resort,  for  military  specifications.  [Ref. 
21]  As  a  result,  many  military  standards  and 
specifications  were  revoked,  suspended,  deleted,  or 
amended. 

One  mandatory  source  of  reliability  management 
guidance  that  was  canceled  was  DoD  5000.20,  Reliability  and 
Maintainability .  This  directive  established  policies  and 
responsibility  for  the  reliability  and  maintainability  of 
defense  systems,  subsystems,  and  equipment.  It  implemented 
the  principles  of  DoD  Directives  and  Instructions  for  major 
system  acquisition  and  for  test  and  evaluation. 

Another  mandatory  source  of  reliability  management 
guidance  that  was  canceled  was  MIL-STD-785B,  Reliability 
Program  for  Systems  and  Equipment  Development  and 
Production.  This  standard  was  comprised  of  application 
requirements,  reliability  program  tasks,  and  an  application 
matrix  that  provided  guidance  and  rationale  for  the 
selection  of  tasks.  It  encouraged  task  selection  tailoring 
for  each  program.  Selected  tasks  could  be  used  to 

solicit  facts  and  recommendations  from  the 
contractors  on  the  need  for,  and  scope  of,  the 
work  to  be  done  rather  than  requiring  that  a 
specific  task  be  done  in  a  specific  way.  [Ref. 

22] 

It  also  included  increased  emphasis  on  reliability 
engineering  tasks  and  tests  to  prevent,  detect,  and  correct 
"design  deficiencies,  weak  parts,  and  workmanship  defects." 
[Ref.  22]  A  discretionary  handbook  providing  this  guidance 


24 


was  to  be  published^  however^  the  standard  was  canceled 
with  no  superseding  document. 

E.  RELIABILITY  AND  THE  ACQUISITION  LIFECYCLE 

Reliability  requirements  are  found  throughout  the 
acquisition  process.  This  section  attempts  to  outline  some 
of  the  most  important  processes  involving  reliability 
within  the  acquisition  lifecycle^  depicted  in  the  figure 
below . 
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Figure  5.  The  5000  Model  [From  Ref.  23:p, 

1 .  Technology  Opportunities  &  User  Needs 


This  is  a  pre-acquisition  period  during  which  the  user 
needs  are  developed,  and  science  and  technology  and  concept 
development  efforts  are  undertaken  in  the  development  of  a 
material  solution.  The  user  needs  are  expressed  in  a 
Mission  Needs  Statement  (MNS)  that  is  developed  in  response 
to  a  threat.  [Ref.  23:p.  9] 

2 .  Concept  &  Technology  Development 
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Concept  &  Technology  Development  is  separated  by  a 
decision  review  into  Concept  Exploration  and  Component 
Advanced  Development.  Concept  Exploration  is  a  period  of 
concept  studies  to  define  and  evaluate  alternative 
concepts.  The  organization  with  the  mission  need  conducts 
an  Analysis  of  Alternatives  (AoA)  to  identify  possible 
system  alternatives  and  their  sensitivities  to  changing 
assumptions.  This  gives  insight  to  Key  Performance 
Parameters  and  their  contribution  to  operational 
capability.  [Ref.  23:p.  13] 

Alternatives  identified  by  the  AoA  are  refined  through 
cost-performance  tradeoff  studies  to  determine  cost  drivers 
for  system  characteristics.  Studies  may  also  determine 
achievable  reliability  objectives  with  respect  to  cost, 
schedule,  and  performance  for  each  alternative.  Similarly 
fielded  systems  may  also  be  studied  to  determine 
operational  reliability  deficiencies.  From  these  studies, 
the  Combat  Developer  (CBTDEV) ,  as  part  of  the  integrated 
concept  team  (ICT) ,  develops  system  operational 
requirements,  of  which  reliability  should  be  one.  These 
operational  requirements  are  consolidated  in  the 
Operational  Requirements  Document  (ORD)  during  the 
requirements  generation  process.  Requirements  that  are 
considered  critical  to  the  success  of  the  system' s  mission 
are  designated  as  Key  Performance  Parameters  (KPPs) .  The 
operational  requirements  address  operational  effectiveness 
(performance  oriented) ,  as  well  as  operational  suitability 
(supportability  oriented),  of  which  reliability  is  key. 
Reliability  should  be  a  focus  here  to  ensure  that  it  is  not 
unnecessarily  sacrificed  or  ignored.  By  designating  it  as 
a  KPP,  reliability  will  receive  the  emphasis  it  requires. 
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Normally^  reliability  requirements  are  stated  as  a 
quantitative  mission  reliability  and  logistics  reliability 
objective.  [Ref.  23:p.  14] 

This  process  leads  to  the  finalization  of  the  ORD  and 
identification  of  KPPs^  the  Acquisition  Program  Baseline 
(APB) ^  and  lifecycle  cost  estimates. 

At  the  decision  review^  the  Milestone  Decision 
Authority  (MDA)  selects  the  preferred  concept  for 
development  with  available  technologies.  The  program  then 
commences  to  Component  Advanced  Development. 

During  the  Component  Advanced  Development  period^  a 
concept  for  the  required  capability  exists^  however  the 
architecture  is  still  unknown.  When  the  system 
architecture  has  been  identified  and  the  component 
technology  has  been  demonstrated^  the  program  may  proceed 
into  the  next  phase.  [Ref.  23:p.  15] 

3 .  System  Development  and  Demonstration 

The  objective  during  this  phase  is  to 

develop  a  system^  reduce  program  risk^  ensure 
operational  supportability ^  design  for 
producibility ^  ensure  affordability^  ensure 
protection  of  Critical  Program  information^  and 
demonstrate  system  integration^  interoperability 
and  utility.  [Ref.  23:p.  17] 

This  is  accomplished  through  the  use  of  simulation-based 
acquisition  and  test  and  evaluation  which  are  guided  by  a 
system  acquisition  strategy  and  test  and  evaluation  master 
plan  (TEMP) . 

This  phase  is  divided  into  two  approaches^  System 
integration  and  System  Demonstration.  At  entry  into  System 
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Integration^  a  system  architecture  exists  but  components 
have  not  been  integrated  into  a  complete  system.  When 
integration  has  been  demonstrated  with  prototypes  and 
system  configuration  is  documented^  the  system  may  enter 
System  Demonstration.  During  System  Demonstration^  the 
system  is  demonstrated  in  its  operational  environment  and 
may  enter  the  next  phase  if  it  meets  the  requirements  and 
commercial  capabilities  are  available.  [Ref.  23:p.  17] 

During  this  phase^  the  system  engineering  process  is 
used  iteratively  to  translate  operational  requirements  into 
a  system  configuration  and  integrate  reliability  and  other 
supportability  factors  to  meet  cost^  schedule^  and 
performance  objectives.  Reliability  Engineers  complete 
analysis  such  as  failure  modes^  fault  tree^  and  parameter 
design  with  designers  and  other  engineers  to  determine 
achievability  of  operational  readiness  and  supportability. 
Reliability  verification  through  testing  programs  are  also 
used  to  verify  the  design  reliability. 

4 .  Production  and  Deployment 

In  this  phase^  the  system  is  approved  for  Low  Rate 
Initial  Production  (LRIP)  during  which  a  limited  number  of 
systems  are  produced  to  establish  the  manufacturing 
capability.  Reliability  testing  programs  are  also  emplaced 
to  ensure  reliability  levels  are  maintained  during 
production.  Failure  prevention  and  corrective  actions  are 
emplaced  to  determine  product  or  process  improvements. 
Production  representative  articles  are  tested  to  determine 
operational  effectiveness  and  suitability  during  Initial 
Operational  Test  and  Evaluation  (lOT&E)  and  Live  Fire  Test 
and  Evaluation  (LFT&E) .  Upon  successful  completion  of 
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testing,  the  system  is  approved  for  Full  Rate  Production 
(FRP)  and  deployment  to  the  using  units.  [Ref.  23 :p.  23] 

5 .  Operations  and  Support 

Operations  and  Support  is  divided  into  two  parts. 
Sustainment  and  Disposal.  During  Sustainment,  the  system 
is  supported  with  a  program  meeting  operational  support 
requirements  in  the  most  cost-effective  manner.  During 
this  phase,  warranties,  such  as  reliability  improvement 
warranties,  may  be  used  to  ensure  continuous  improvement  of 
the  fielded  system  through  engineering  change  proposals  to 
faulty  components.  Additionally,  reliability  centered 
maintenance  may  be  implemented  to  identify  reliability 
degrading  trends  and  determine  root  cause.  Upon  reaching 
the  end  of  its  useful  life,  the  system  is  disposed  of,  but 
not  after  possible  service  life  extension  or  rebuild 
programs.  [Ref.  23:p.  27] 

F.  SUMMARY 

High  reliability  is  an  important  and  necessary 
requirement  for  weapon  system  development.  It  can  reduce 
the  logistics  burden  felt  by  the  services  and  also  reduce 
lifecycle  costs,  freeing  funds  for  the  procurement  of  new 
equipment  to  modernize  the  force. 


However^ 

its 

achievement 

can 

be 

elusive  for  many 

weapon 

system 

programs .  The 

FCS 

is 

one  weapon  system 

program 

currently 

in  development 

that 

is  attempting  to 

achieve  high  or  ultra-reliability  to  reduce  the  logistics 
burden . 

Both  mandatory  and  discretionary  guidance  for 
establishing  and  achieving  reliability  requirements  exists, 
however,  there  is  not  as  much  as  there  once  was.  Also, 
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once  mandatory  military  standards  and  specifications  have 
been  canceled  or  made  discretionary  in  favor  of  commercial 
practices . 

Because  reliability  is  encountered  throughout  the 
lifecycle,  it  is  necessary  to  effectively  and  efficiently 
manage  it.  Therefore,  an  exploratory  investigation  is 
necessary  to  determine  what  successful  programs  are  doing 
to  achieve  reliability  requirements. 
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III.  METHODOLOGY 


A.  INTRODUCTION 

This  chapter  outlines  the  methodology  used  for  the 
collection  of  data.  It  includes  the  purpose  of  the  study, 
the  research  method,  the  questions  asked,  and  the  selection 
of  study  subjects  and  their  description. 

B .  PURPOSE 

The  purpose  of  this  exploratory  study  was  to  determine 
what  successful  programs  implemented  to  achieve  reliability 
requirements  given  that  more  than  80%  of  programs  fail. 
The  objective  of  the  study  was  to  identify  successful 
practices,  pitfalls,  and  recommendations  for  process 
improvement.  The  study  focused  on  those  successful 

programs  that  recently  achieved  reliability  requirements 
and  those  people  who  were  intimately  involved  with  the 
achievement  of  reliability  requirements. 

C .  EXPLORATORY  STUDY 

1 .  Study  Overview 

The  primary  objective  of  the  interviews  was  to 
identify  how  successful  programs  achieved  reliability 
requirements.  Programs  that  were  successful  in  achieving 
reliability  requirements  were  identified,  and  a  person 
familiar  with  the  program' s  reliability  requirements  was 
then  identified  and  questioned  about  program  practices. 

2 .  Interview  Question  Development 

Questions  evolving  from  the  secondary  research 
questions  were  developed  to  determine  the  successful 
strategies  or  practices  used  by  programs  to  achieve 
reliability  requirements.  The  focus  of  these  questions  was 
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to  identify  the  program's  organization,  how  requirements 
were  established,  managed,  and  achieved,  and  when  the 
program  focused  on  reliability.  Additional  questions  were 
asked  to  gather  lessons  learned  and  recommendations  for 
improving  the  process.  Lastly,  questions  were  posed  to 
determine  if  ultra-reliability  was  achievable  and 
worthwhile.  The  interview  questions  are  found  in  Appendix 
A. 

The  first  series  of  questions  were  developed  to 
determine  the  organization  of  the  programs  to  achieve  their 
objective  requirements.  The  questions  asked  how  the 
program  was  organized  for  reliability  to  determine  if  an 
Integrated  Product  Team  was  used,  and  who  primarily  was 
responsible  for  reliability  issues  within  the  program. 

The  second  series  of  questions  were  intended  to 
determine  the  development  of  reliability  requirements.  The 
questions  asked  if  the  program  had  any  input  in 
establishing  reliability  requirements,  how  they  were 
determined,  and  what  the  requirements  according  to  the 
Operational  Requirements  Document  were,  how  they  were 
worded,  and  how  they  were  conveyed  to  the  contractor. 

The  third  series  of  questions  were  developed  to 
determine  how  the  programs  managed  reliability.  The 
questions  asked  how  the  contractor  managed  reliability 
growth,  and  what  oversight  the  program  office  had.  Other 
questions  asked  if  the  program  had  a  Reliability  Program 
Plan,  what  its  key  elements  were  and  if  their  approach  was 
varied.  The  last  question  asked  what  primary  sources  of 
reliability  guidance  were  used  and  if  there  was  adequate 
guidance . 
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The  fourth  series  of  questions  were  developed  to 
determine  what  processes  programs  used  to  achieve 
reliability  requirements  and  how  this  was  measured.  The 
first  question  asked  if  reliability  was  a  Key  Performance 
Parameter  (KPP)  to  determine  the  relative  importance.  The 
next  questions  asked  what  level  of  reliability  was  achieved 
and  how  successful  the  program  considered  itself  and  why. 
The  next  questions  asked  how  reliability  growth  was 
measured^  how  testing  was  performed^  what  the  key  source  of 
unreliability  was^  and  how  developmental  reliability 
figures  compared  with  operational  figures.  The  last 
questions  asked  if  the  contractor  was  incentivized  in  any 
way  and  how  the  program  planned  to  maintain  the  system' s 
reliability  once  fielded. 

The  fifth  series  of  questions  were  developed  to 
determine  when^  in  the  acquisition  lifecycle^  programs 
focused  on  reliability.  Also  asked  were  some  of  the  most 
important  steps  or  processes  within  the  acquisition 
lifecycle . 

The  sixth  series  of  questions  were  intended  to  gather 
lessons  learned^  recommendations  for  improvement^  and 
pitfalls  to  avoid. 

The  last  series  of  questions  were  developed  to 
determine^  in  the  opinion  of  the  subject^  if  ultra¬ 
reliability  was  achievable  and  worthwhile. 

3.  Identification  of  Study  Subjects 

Study  participants  were  identified  by  the  Director  of 
Reliability  and  Maintainability^  Army  Test  and  Evaluation 
Command  (ATEC) .  Selection  of  the  study  participants  was 
based  on  successful  or  near  successful  completion  of 


33 


Operational  Testing  with  respect  to  reliability.  The 
reason  successful  programs  were  chosen  is  twofold.  Firsts 
successful  programs  would  better  understand  how  to  achieve 
reliability  requirements  than  unsuccessful  programs. 
Second^  successful  programs  would  be  more  willing  to  share 
the  reasons  for  their  success.  Of  the  more  than  140 
programs  for  which  ATEC  has  visibility^  only  ten  programs 
were  identified  as  successful.  [Ref.  24]  These  ten  programs 
included:  Abrams  System  Enhancement  Program  (SEP) ^  Bradley 
A3^  Bradley  Fire  Support  Team  (BFIST) ^  M270A1  Launcher^ 
Family  of  Medium  Tactical  Vehicles  (FMTV) ,  High  Mobility 
Multipurpose  Wheeled  Vehicle  (HMMWV) ,  Aviation  Mission 
Planning  System  (AMPS) ,  Long  Range  Advanced  Sensor  System 
Suite  (LRASS)^  Modular  Crowd  Control  Munition  (MCCM) ^  and 
Target  Point  Illuminator  Aiming  Light  (TPIAL) .  Of  the  ten 
programs^  two^  the  MCCM  and  TPIAL^  had  no  Reliability 
Engineers  or  reliability  programs  because  of  the  design 
simplicity  and  non-developmental  nature  of  the  programs^ 
and  one^  M270A1  Launcher^  could  not  be  reached  for 
questionning .  After  contacting  the  programs^  two  more 
programs  were  added  to  the  list  because  of  successful 
reliability  achievement.  These  two  programs  are  Driver's 
Vision  Enhancement  (DVE)  and  Tactical  Weapons  Sight  (TWS) . 
A  brief  description  with  reliability  requirements  and 
achievements  of  each  of  the  nine  programs  questioned 
follows . 

a.  Abrams  System  Enhancement  Package  (SEP) 

Abrams  SEP  upgrades  the  M1A2  Abrams  tank  by 
adding  second-generation  thermal  sensors^  Embedded  Battle 
Command  (EBC)  command  and  control  software^  a  Temperature 
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Management  System  (TMS) ,  and  an  Under  Armor  Auxiliary  Power 
Unit  (UAAPU) ,  among  others.  [Ref.  25] 


The  Abrams  SEP  Program  conducted  Follow-on 
Operational  Test  and  Evaluation  (FOT&E)  IV  in  November 
2000  .  The  system  had  a  requirement  for  101  Mean  Miles 
Between  Failure  (MMBF)  and  demonstrated  511  MMBF.  The 
system  also  required  a  combat  reliability  requirement  of 
320  MMBF  and  demonstrated  881  MMBF. 

ib.  Bradley  A3 

The  Bradley  A3  integrates  the  Second  Generation 
Forward  Looking  Infrared  in  the  Improved  Bradley 
Acquisition  System  (IBAS)  sight  and  Commander's  Independent 
Viewer  (CIV) ,  automated  ballistic  solutions  and  target 
tracking  software.  [Ref.  26] 

The  Bradley  A3  Program  conducted  Initial 
Operational  Test  and  Evaluation  (lOT&E)  in  the  fall  of 
2000.  The  system  had  a  requirement  for  400  Mean  Miles 
Between  Failure  (MMBF)  and  demonstrated  417  MMBF. 

c.  Bradley  Fire  Support  Vehicle  (BFIST) 

The  BFIST  integrates  the  Second  Generation 
Forward  Looking  InfraRed,  Commander' s  Independent  Viewer 
(CIV) ,  Improved  Fire  Control,  and  Fire  Support  Vehicle 
Mission  Equipment  Package  onto  a  Bradley.  [Ref.  27] 

The  BFIST  Program  had  a  reliability  requirement 
of  675  MMBF.  The  program  conducted  a  Production 
Qualification  Test,  demonstrating  992  MMBF  and  a  Production 
Verification  Test,  demonstrating  1051  MMBF. 
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d.  Family  of  Medium  Tactical  Vehicles  (FMTV) 

The  FMTV  Program  uses  a  common  truck  chassis  for 
several  vehicle  configurations  in  two  payload  classes  (2.5 
ton  and  5  ton)  and  two  tactical  trailers  with  complementary 
payloads.  [Ref.  28:p.  281] 

The  FMTV  Program  recently  underwent  Production 
Verification  Testing  in  January  1999.  Reliability 
requirements  and  achievements  are  depicted  in  the  table 
below . 


Requirement  (MMBHMF) 

Achievement  (MMBHMF) 

LMTV  Cargo 

5500 

13333 

LMTV  Van 

4200 

6667 

LMTV  Trailer 

2800 

20000 

MTV  Cargo 

5500 

19048 

MTV  Tractor 

3800 

6667 

MTV  Wrecker 

2800 

4000 

MTV  Trailer 

2600 

20000 

Table  4.  FMTV  Reliability  Requirements/Achievements 

e.  High  Mobility  Multi-purpose  Wheeled  Vehicle 
(HMMWV) 

The  HMMWV  Program  is  a  tri-service  program  using 
a  light,  highly  mobile,  diesel-powered,  four-wheel-drive 
vehicle  with  a  common  chassis  for  several  configurations. 
A2  improvements  include  a  four-speed,  electronic 
transmission,  a  6.5-liter  diesel  engine,  and 
transportability  improvements.  [Ref.  28 :p.  265] 

The  HMMWV  Program' s  recent  test  results  are 
depicted  in  the  table  below. 
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HMMWV  Carrier 

Requirement  (MMBHMF) 

Achievement  (MMBHMF) 

Cargo 

1900 

4292 

TOW/Armament 

1800 

2937 

Shelter 

1600 

2947 

Ambulance 

1600 

1667 

Table  5.  HMMWV  Reliability  Requirements/Achievements 

f.  Aviation  Mission  Planning  System  (AMPS) 

AMPS  is  a  Personal  Computer  based  subordinate 
system  of  the  Army  Battle  Command  System,  Maneuver  Control 
System.  AMPS  facilitates  aviation  planning  functions  and 
automates  brigade  and  below  planning  and  distribution  of 
mission  files.  [Ref.  29] 

AMPS  recently  underwent  Operational  Testing  in 
March  and  April  2001.  The  system  had  a  requirement  for  154 
hours  Mean  Time  Between  Mission  Affecting  Failure  (MTBMAF) 
and  demonstrated  233.3  hours  MTBMAF. 

g.  Long  Range  Advanced  Scout  Surveillance 
System  (LRAS3) 

LRAS3  consists  of  a  second  generation  Forward 
Looking  Infrared  with  long-range  optics,  eye-safe  laser 
rangefinder,  a  day  video  camera,  and  a  Global  Positioning 
System  (GPS)  operating  in  both  mounted  and  dismounted 
configurations.  [Ref.  28:p.  105] 

The  system  had  a  requirement  for  330  hours  Mean 
Time  Between  Essential  Function  Failure  (MTBEFF)  and 
demonstrated  364  hours  MTBEFF. 

h.  Driver's  Vision  Enhancement  (DVE) 

The  DVE  is  a  passive,  un-cooled  thermal-imaging 
system  for  drivers  of  combat  and  tactical  wheeled  vehicles . 
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The  sensor  module  consists  of  a  second-generation  Forward 
Looking  Infrared  device  that  delivers  a  video  image  to  a 
high-quality,  commercial,  flat-panel  display  and  control 
module.  [Ref.  30] 

The  system  had  a  requirement  for  740  hours  Mean 
Time  Between  Operational  Mission  Failure  (MTBOMF)  and 
demonstrated  776  hours  MTBOMF. 

i.  Tactical  Weapon  Sight  (TWS) 

The  AN/PAS-13  TWS  is  a  family  of  low-cost, 
lightweight,  infrared  imaging  devices  using  second- 
generation  Forward  Looking  Infrared  with  medium  to  high 
resolution.  The  system  provides  a  standard  video  output 
for  training,  image  transfer  or  remote  viewing  and  is  used 
for  fire  control  of  individual  and  crew-served  weapons. 
[Ref.  30] 

The  system  had  a  requirement  for  250  hours  MTBOMF 
and  demonstrated  3513  hours  MTBOMF. 

4 .  Interviews 

Interviews  were  scheduled  for  the  last  two  weeks  of 
February  and  the  first  week  of  March  2002.  Introductory 
telephone  calls  were  made  prior  to  e-mailing  letters  of 
introduction  with  questionnaire  attachments.  Appendix  B 
contains  a  sample  e-mail  message  with  a  letter  of 
introduction.  Study  subjects  were  offered  the  option  to 
answer  the  questionnaire  by  e-mail  or  by  telephone. 

D.  SUMMARY 

The  methodology  for  collecting  the  data  was  to 
identify  successful  programs  and  those  people  within  the 
program  who  had  intimate  knowledge  of  reliability  issues. 

These  people  were  questioned  regarding  their  practices  in 
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achieving  reliability  requirements,  lessons  learned. 


and 


thoughts 

interviews 


on  ultra-reliability.  The  results  of  the 

are  presented  in  the  next  chapter. 
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IV.  DATA  SUMMARY 


A.  INTRODUCTION 

This  chapter  presents  the  subjective  data  gathered 
from  the  interviews.  Data  is  summarized,  presenting  common 
trends  and  differences.  Questions  and  responses  are 
grouped  by  subject  area:  Organizing  for  reliability; 
requirement  development;  reliability  management; 
reliability  achievement;  reliability  timeline;  lessons 
learned/recommendations  for  improvement;  and  ultra¬ 
reliability  . 

B .  INTERVIEW  RESULTS 

A  total  of  seven  interviews  involving  nine  programs 
were  conducted  as  scheduled  with  two  telephonic  interviews 
on  2  6  February  and  4  March  2002,  and  the  remainder  by  e- 
mail.  Results  are  presented  below.  Respondents  are  not 
identified  to  protect  anonymity. 

1 .  Organizing  for  Reliability 

The  objective  of  these  questions  was  to  determine  if 
programs  used  dedicated  Reliability  Integrated  Product 
Teams  (IPTs)  and  who,  within  the  program,  was  primarily 
responsible  for  reliability. 

Eight  of  nine  programs  stated  that  the  Test 
Integration  Working  Group  (TIWG)  was  primarily  responsible 
for  developing  reliability  requirements,  while  only  one 
program  had  a  Reliability  IPT  for  this.  A  Reliability, 
Availability,  Maintainability  (RAM)  integrated  working 
group  was  incorporated  subordinately  into  the  TIWG.  This 
group  developed  the  Failure  Definition  and  Scoring  Criteria 
(FDSC)  and  "crosswalked  and  confirmed  requirements  with  the 
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ORD".  The  TIWG  was  also  primarily  responsible  for 
developing  the  test  plan  for  the  program.  The  TIWG  was 
composed  of  representatives  from  the  test  and  evaluation 
community^  the  material  developer^  and  the  user.  The  TIWG 
met  at  least  quarterly  initially^  then  more  often^  usually 
monthly  and  bimonthly^  just  prior  to  and  during  tests. 

Primary  responsibility  for  reliability  within  the 
programs  was  assigned  to  a  reliability  and  maintainability 
branch  chief  in  two  cases.  in  most  other  programs^  a 
Reliability  Engineer^  most  often  provided  by  matrix 
support^  was  responsible  for  program  reliability.  Of  the 
few  programs  that  did  not  have  Reliability  Engineers^ 
Mechanical  Engineers  were  primarily  responsible  for 
reliability.  These  representatives  were  most  often 
supported  by  the  TIWG  and  were  members  of  the  RAM  scoring 
and  assessment  conferences. 

2 .  Requirement  Development 

The  objective  of  these  questions  was  to  determine  if 
programs  had  influenced  requirement  establishment. 

in  all  cases^  a  program  reliability  representative  was 
able  to  provide  input  for  establishing  reliability 
requirements.  This  was  most  often  accomplished  through  the 
use  of  a  RAM  Rationale  Report.  The  user  representative 
would  develop  the  requirements  based  on  what  was  considered 
as  the  need.  The  program  reliability  representative  then 
analyzed  the  requirement  considering  ''if  industry  may  be 
able  to  meet  those  requirements  with  [the  technology] 
industry  has  available  today."  if  analysis  supported  the 
requirement^  and  it  was  acceptable  to  both  the  user  and 
program  representative^  it  remained  unchanged. 
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Reliability  requirements  for  all  programs  were 
primarily  based  on  historical  data  from  similar  previous 
systems,  taking  into  consideration  "state  of  the  art  and 
technical  feasibility."  In  one  case,  "200,000  miles  of 
historical  RAM  maturity"  data  was  used.  Other  methods 
included  "leveraging  similar  component  reliability  using 
other  similar  programs,"  and  the  "Duane  modeling 
methodology."  In  another  case,  data  from  the  Rome  Analysis 
Center's,  Reliability  Engineer's  Toolkit  and  manufacturer's 
specifications  were  also  considered  in  developing 
requirements . 

These  requirements,  outlined  in  the  previous  chapter, 
were  included  in  contract  performance  specifications. 
Acquisition  Program  Baselines,  and  Milestone  III  Exit 
Criteria.  In  most  cases,  the  reliability  requirements  were 
worded  as  Mean  Miles  Between  Failure  (MMBF)  for  mechanical 
systems  or  Mean  Time  Between  Essential  Function  Failure 
(MTBEFF)  for  electrical  systems.  In  two  cases,  the 
reliability  requirements  from  the  Operational  Requirements 
Document  were  expressed  in  terms  of  Mean  Miles  Between 
Operational  Mission  Failure  (MMBOMF) .  It  was  necessary  to 
express  these  requirements  as  functions  of  hardware  in 
terms  of  Mean  Miles  Between  Hardware  Mission  Failure 
(MMBHMF)  so  that  the  contractor  could  be  held  responsible 
for  sytem  performance.  To  make  the  conversion  between 
MMBOMF  and  MMBHMF,  a  historical  "rule  of  thumb"  was  used, 
increasing  the  operational  requirement  by  one-third  to 
arrive  at  the  system  requirement. 

3 .  Reliability  Management 
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The  objective  of  these  questions  was  to  determine  if 
programs  implemented  a  Reliability  Program  Plan. 

In  two  cases,  programs  began  development  and  testing 
before  implementation  of  specification  reform.  This 
reform  prevented  specifying  'how  to'  design,  manufacture  or 
test  for  reliability  to  contractors.  One  program  stated 
that  the  reason  for  its  success  was  because  it  was  started 
prior  to  specification  reform. 

Another  program  stated  that  AR  702-3  was  the 
controlling  document  for  its  program.  The  contractor 
prepared  a  RAM  design  guidebook  to  be  used  in  all  designs 
and  "detailed  documentation  along  with  approved  contractor 
program  plans  allowed  for  oversight  and  review  of  the 
contractor  progress." 

One  program  used  "Tiger  Teams"  of  contractor 
representatives  to  conduct  root  cause  analysis  for  "hot 
issues."  Additionally,  a  Corrective  Action  Management 
Review  (CAMR) ,  which  had  representatives  from  both  the 
program  and  contractor,  periodically  conducted  reviews  to 
answer  systemic  reliability  issues. 

Another  program' s  prime  contractor  was  a  Government 
agency,  which  already  had  a  proven  reliability.  Oversight 
was  accomplished  through  design  reviews  and  configuration 
control  meetings . 

Only  three  programs  had  Reliability  Program  Plans 
prepared  for  the  program  office.  Two  others  had  plans 
prepared,  which  the  contractor  followed.  The  remaining 
programs  that  didn't  have  plans,  stated  that  they  weren't 
necessary  because  of  the  limited  developmental  nature  of 
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the  program  or  because  of  improvement  upon  a  legacy  system. 
One  legacy  system  program  had  used  Reliability  Program 
Plans  in  the  past  that  included  Failure  Modes,  Effects  and 
Criticality  Analysis  (FMECA) ,  reliability  growth  curves, 
corrective  actions  systems,  and  other  key  metrics. 

Of  those  programs  that  had  Reliability  Program  Plans, 
most  of  the  plans  were  followed  closely  to  ensure  the 
contractor  was  on  the  "right  track."  Some  plans  may  have 
varied  based  upon  reliability  issues  such  as  failures  and 
corrective  actions,  or  to  consider  value-adding  changes. 

The  primary  sources  of  Government  guidance  used 
consisted  of  both  canceled  and  current  publications.  The 
sources  included  MIL-STD-781  and  MIL-HDBK-781 ,  Reliability 
Testing  for  Engineering  Development ,  Qualification ,  and 
Production,  MIL-HDBK-189,  Reliability  Growth  Management,  AR 
702-3,  Army  Materiel  Systems  Reliability,  Availability,  and 
Maintainability  (RAM),  TRADOC  Pamphlet  70-11,  RAM  Rationale 
Handbook ,  AR  70-1,  Army  Acquisition  Policy  and  MIL-STD- 
1629,  Procedures  for  Performing  a  Failure  Mode,  Effects  and 
Criticality  Analysis .  Additional  information  was  gathered 
from  other  military  standards,  best  practices  and  meetings 
with  experts  in  the  field. 

Some  respondents  felt  there  was  not  enough  guidance 
and  "we  went  from  one  extreme  to  the  other  in  specification 
reform."  Another  believed  it  was  now  necessary  to  develop 
reliability  policies  and  ensure  they're  enforced. 

4 .  Reliability  Achievement 

The  objective  of  these  questions  was  to  determine  what 
processes  programs  used  to  achieve  reliability  requirements 
and  how  this  was  measured. 
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Reliability  requirements  were  a  Key  Performance 
Parameter  (KPP)  for  only  one  program.  In  another  case,  it 
was  stated  that  reliability  exceeded  the  KPP  level  and 
received  4-star  (general  officer)  visibility.  The  other 
programs  agreed  that  reliability  was  important  and 
"essential  to  fielding  an  acceptable  system  to  the 
soldiers" . 

Two  programs  rated  their  reliability  achievements 
(outlined  in  the  previous  chapter)  as  extremely  successful 
because  they  exceeded  their  requirement.  The  reasons 
stated  for  their  success  were  due  to  early  establishment  of 
FDSC,  well  managed  scoring  conferences,  working  with  the 
user  and  RAM  community,  funded  and  continuous  well  managed 
tests,  and  an  effective  failure  analysis  process  as  well  as 
processing  of  Engineering  Change  Proposals. 

Two  programs  rated  themselves  as  moderately  successful 
because  they  exceeded  the  requirement,  but  were  plagued 
with  software  or  hardware  problems. 

The  remaining  five  programs  rated  themselves  as 
average  because  of  difficulty  in  achieving  requirements  and 
also  because  they  are  continuing  to  mature  the  program.  One 
program  indicated  its  primary  reason  for  success  was  due  to 
"hard  requirements"  that  had  to  be  met  due  to  the  program's 
visibility.  The  program  also  said  "top  management"  must 
tout  the  importance  of  reliability  achievement. 

The  use  of  design  analysis  tools  or  processes  for  root 
cause  analysis  varied  by  program.  One  respondent  stated 
that  the  program  "didn't  work  to  influence  design"  because 
of  the  integration  of  non-developmental  items .  The  program 
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only  relied  on  the  test-fix-test  process  during  the  test 
phase . 

One  program  used  Failure  Reporting  and  Corrective 
Action  System  (FRACAS)  to  quickly  analyze  failures  and  make 
corrections.  The  program  used  site  contractor 
representatives  to  provide  more  timely  data  than  Government 
systems  were  capable  of. 

Another  program  used  a  fishbone  cause  and  effect 
process,  FMECA,  Sneak  Circuit  Analysis,  fault  tree,  and 
component  level  Environmental  Stress  Screening  (ESS) , 
Highly  Accelerated  Life  Testing  (HALT) ,  and  Highly 
Accelerated  Stress  Testing  (HAST)  as  its  primary  design 
analysis  tools. 

To  manage  design  analysis,  three  programs  created  a 
database  to  track  failures  during  testing  and  to  conduct 
root  cause  analysis  with  technical  engineers  from  both  the 
Government  and  contractor. 

Both  developmental  and  operational  testing  played  a 
major  role  in  determining  reliability  achievement  for  all 
programs.  All  programs  used  many  iterations  of  test-fix- 
test  during  developmental  tests  prior  to  operational 
testing.  Three  programs  also  used  Limited  User  Tests  as  a 
prelude  to  Operational  Testing.  Additionally,  due  to  the 
maturity  of  their  program,  three  programs  used  Production 
Qualification  Tests  (PQT)  and  Production  Verification  Tests 
(PVT)  to  ensure  quality  and  verify  the  production  line. 

In  comparing  reliability  figures  from  Developmental 
Testing  to  Operational  testing,  most  programs  agreed  that 
there  was  a  reduction  due  to  environment,  maintenance,  and 
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"operator  induced  failures"  although  in  some  cases,  it  was 
hard  to  measure,  especially  when  testing  software.  One 
program  experienced  only  a  10-15%  reduction,  while  two 
others  experienced  a  20-30%  reduction. 

Key  sources  of  reliability  problems  were  cited  as 
mostly  due  to  hardware,  although  some  software  problems 
were  noted.  Most  hardware  problems  were  a  result  of  the 
environment  due  to  water  and  mud  infiltration  of  seals, 
vibration,  and  extreme  temperature  variation.  Software  was 
also  a  problem,  and  in  one  case,  the  "software  architecture 
was  not  robust  and  did  not  lend  itself  to  continuous  change 
and  upgrade . " 

In  only  two  cases,  the  contractor  was  incentivized  to 
achieve  reliability  requirements.  This  was  not  implemented 
for  the  entire  system,  but  only  for  "troublesome 
components...  to  assure  getting  required  life  with  a  bonus 
for  life  beyond  requirements."  Most  other  programs 
considered  incentivizing  the  contractor  to  be  unnecessary 
because  the  contractor  was  required  to  achieve  requirements 
in  the  contract.  For  one  program,  "continuance  of  the 
program  was  the  incentive,  it  either  met  reliability  or 
there  wasn't  a  program." 

In  planning  for  maintaining  the  system' s  reliability 
after  fielding,  one  respondent  "hadn't  thought  about  that." 
This  respondent  believed  that  once  reliability  requirements 
were  achieved,  it  was  unnecessary  to  try  to  maintain  that 
level  unless  major  engineering  changes  were  made  to  the 
system.  If  engineering  changes  were  made,  then  it  would  be 
necessary  to  test  to  check  reliability  and  use  the  test- 
fix-test  process.  It  was  also  thought  that  because  the 
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operating  environment  couldn't  be  controlled,  collection  of 
field  data  was  unnecessary  and  couldn't  be  incorporated 
into  the  program  because  it  was  inaccurate.  The  respondent 
went  on  to  state  that  logisticians  are  more  interested  in 
this  information  and  will  make  appropriate  recommendations 
to  the  Program  Manager  for  reliability  improvement. 

Another  program  uses  Original  Equipment  Manufacturer 
(OEM)  system  service  representatives  at  major  installations 
where  the  system  is  fielded.  These  contractor  service 
representatives  are  similar  to  Army  Materiel  Command  (AMC) 
Logistics  Assistance  Representatives.  They  "provide 
technical  expertise  for  correcting  problems,  locate  parts, 
and  suggest  engineering  change  proposals."  The  difference 
is  that  service  representatives  are  specific  to  only  one 
type  of  system  and  are  therefore,  more  knowledgeable, 
especially  for  newly  fielded  systems.  They  also  gather 
availability  data  and  analyze  it  for  the  source  of  downtime 
in  an  attempt  to  improve  it. 

Another  program  used  a  Field  Problem  Review  board  to 
analyze  field  data  and  "decide  what  problems  to 
investigate,  fix,  and  retrofit  in  the  field."  The  program 
would  also  periodically  test  a  production  system  to  ensure 
production  systems  still  met  reliability  requirements. 

A  similar  support  plan  was  used  by  another  program. 
This  program  used  a  System  Assessment  Management  program 
and  Corrective  Action  Management  Reviews  (CAMR)  to  ensure 
reliability  after  fielding. 

Another  program  planned  to  "monitor  the  system' s 
reliability  after  fielding  via  Deficiencies  Product  Reports 
(DPR)  and  Contractor  Logistics  Support  (CLS) 
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One  last  program  believed  the  Army  does  not  adequately 
"track  reliability  for  systems  once  they  are  fielded."  The 
Army  maintenance  system  is  not  equipped  to  "capture 
valuable  data  that  could  be  used  to  formulate  reliability 
requirements  for  future  vehicles."  "Lessons  learned  in  the 
field  could  be  applied  to  current  procurements  to  save  time 
and  money  in  the  future." 

5 .  Reliability  Timeline 

The  objective  of  these  questions  was  to  determine  when 
to  focus  on  reliability  and  what  some  of  the  most  important 
steps  and  processes  within  the  lifecycle  were . 

All  programs  began  focusing  on  reliability  early  in 
the  acquisition  lifecycle.  Most  programs  agreed  that  from 
"day  one,"  reliability  should  be  a  key  focus  of  the 
program.  It  was  essential  for  the  program  to  be  involved 
in  establishing  reliability  requirements  and  getting 
leadership  to  "support  the  reliability  program  requirements 
and  insure  they  will  be  met." 

Four  programs  said  that  reliability  must  be  a  key 
focus  during  System  Development  and  Demonstration  (SDD) 
phase  of  the  lifecycle. 

Some  responses  for  the  most  important  steps  affecting 
reliability  were  outlined  as  beginning  with  establishing 
requirements.  It  is  important  to  ensure  "early,  accurate 
analysis  of  requirements"  and  that  they  are  "agreed  to  and 
understood  by  all  as  early  as  possible."  Next  was  the 
development  of  Failure  Definition  and  Scoring  Criteria  to 
clearly  outline  failures  for  analyzing  test  results. 
Followed  by  steps  to  influence  design  using  and  to  predict 
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or  estimate  reliability.  Lastly,  was  test-analyze-test  to 
ensure  reliability  achievement. 

6 .  Lessons  Learned/Recommendations  for  Improvement 

The  objective  of  these  questions  was  to  identify  any 
lessons  learned  from  the  program' s  attempts  to  achieve 
reliability  requirements  and  to  also  identify 
recommendations  for  improvement  to  the  acquisition  process 
regarding  reliability. 

Responses  varied  for  steps  or  processes  that  should  be 
changed.  One  program  indicated  the  Army  maintenance  system 
should  be  responsible  for  collecting  failure  data.  This 
would  allow  Reliability  Engineers  to  use  this  data  to 
establish  requirements  for  future  components  or  systems. 
Similarly,  another  program  thought  a  more  active  role  was 
required  in  component  decisions  and  better  planning  was 
needed  for  availability  of  spares. 

Another  program  indicated  that  some  specifications  and 
standards  should  be  reinstated  which  was  reinforced  by 
another  program  that  thought  the  contractor  should  be 
required  to  use  reliability  data  gathered  from  the  field  in 
designing  the  system.  Two  programs  indicated  that  the 
Government  should  be  more  stringent  in  enforcing  contractor 
achievement  of  reliability  requirements. 

A  last  program  wanted  to  "allow  probability  range  on 
combat  mission  failures."  This  program  also  indicated  that 
the  programs  should  use  "technical  versus  operational 
testing."  It  also  thought  that  "participation  of  the 
system  contractor"  should  be  allowed  in  operational 
testing . 
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Suggestions  for  other  programs  to  achieve  reliability 
requirements  were  wide  ranging.  Most  programs  did  agree 
that  it  was  necessary  to  "get  management  to  focus  on 
reliability."  A  majority  of  the  programs  also  agreed  that 
it  is  necessary  to  ensure  effective  communications  between 
the  program,  user,  contractor,  and  RAM  community.  Other 
suggestions  were  to  establish  a  Reliability  IPT,  "get 
involved  in  the  reliability  program  at  the  beginning, "  and 
follow  up  on  the  reliability  status  on  a  monthly  basis. 
Early,  accurate  requirements  establishment,  analysis  and 
traceability  and  ability  to  influence  design  were  also 
important.  General  consensus  existed  for  establishing 
detailed  Failure  Definition  and  Scoring  Criteria  to  ensure 
proper  assessment  of  tests.  It  is  also  necessary  to  "test 
early  and  often,"  and  effectively  analyze  and  correct 
failures.  For  software  development,  a  program  suggested 
use  of  "open  architecture  and  object  oriented  design"  and 
use  of  a  Software  Engineering  Institute  (SEI)  Capability 
Maturity  Model  (CMM)  level  3  or  higher  software  design 
organization . 

Common  pitfalls  to  be  avoided  were  varied  as  well. 
One  program  warned  against  establishing  "unrealistic 
requirements,  an  unrealistic  growth  curve  based  on 
resources  and  time,"  and  poor  definition  of  FDSC .  One 
respondent  indicated  that  a  lack  of  communication  and 
support  from  management  must  be  avoided. 

7 .  Ultra-Reliability 

The  last  series  of  questions  were  meant  to  determine 
if  ultra-reliability  should  be  a  Key  Performance  Parameter 
(KPP)  and  if  it  is  achievable . 
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Responses  were  mixed  regarding  whether  ultra¬ 
reliability  should  be  a  KPP.  At  least  one  respondent  was 
unfamiliar  with  the  term  and  would  not  speculate  on  its 
achievability  or  worth. 

Most  programs  indicated  that  establishing  ultra¬ 
reliability  as  a  KPP  and  achieving  it  was  dependent  on  the 
"system,  its  operating  environment,  and  the  use  of 
commercial  items." 

Mechanically  oriented  programs  were  divided  regarding 
ultra-reliability.  Two  programs  thought  reliability  was  an 
important  consideration  and  may  be  possible  if  enough  time 
and  money  were  invested.  Two  programs  indicated  that 
ultra-reliability  should  not  be  a  KPP  for  systems  such  as 
theirs  because  it  is  not  realistic  or  possible  for 
mechanical  systems.  They  stated  that  ultra-reliability  may 
be  possible  for  "electronics  or  computer  software"  but  is 
"not  realistic"  for  "military  mechanical  systems  pounding 
the  cross  country  terrain  at  40  mph"  and  is  therefore, 
unachievable  without  exotic  materials  and  unrealistic 
subsystem  redundancy. 

Most  electronically  oriented  programs  indicated  that 
ultra-reliability  should  be  a  KPP  because  soldiers  in  the 
field  need  reliable  systems.  One  program  stated  that  it 
should  only  be  considered  where  "safety  is  a  critical 
concern."  They  also  thought  that  it  might  be  achievable  if 
the  program  had  the  money  and  time  necessary  to  invest  in 
its  development. 

C.  SUMMARY 

Seven  respondents  representing  nine  programs  described 
their  successful  strategies  and  practices  for  achieving 
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reliability  requirements .  The  viewpoint  for  the  responses 
was  from  the  perspective  of  the  person  most  responsible  for 
reliability  within  the  program.  The  respondents  described 
their  approach  for  organization,  requirement  development, 
reliability  management,  achievement  and  timeline,  lessons 
learned,  and  thoughts  on  ultra-reliability.  This 
information  will  be  analyzed  in  the  next  chapter, 
identifying  trends  and  differences,  and  comparing  these  to 
guidance . 
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V.  DATA  ANALYSIS 


A.  INTRODUCTION 

This  chapter  summarizes  the  analysis  of  the  subjective 
data  gathered  during  the  interviews  of  the  nine  successful 
programs.  Responses  were  analyzed  by  subject  area^ 
revealing  common  trends  and  differences  in  comparison  to 
published  reliability  guidance^  while  identifying 
advantages  and  disadvantages  of  practices. 

B.  ANALYSIS  OF  SUBJECT  AREAS 

1.  Organizing  for  Reliability 

DA  PAM  70-3  recommends  organizing  a  Reliability 
Integrated  Process  Team  (IPT)  ''to  review^  classify  and 
charge  R&M  data  from  system  level  development  and 
operational  tests."  Participants  of  the  Reliability  IPT 
are  the  Program  Office  or  Materiel  Developer  (MATDEV) ,  the 
user  or  Combat  Developer  (CBTDEV) ^  the  Training  Developer 
(TNGDEV) ^  and  the  independent  evaluator.  It  further  states 
that  "R&M  IPTs  should  be  held  periodically  during  system 
level  testing"  with  a  final  IPT  "held  at  the  conclusion  of 
each  test."  [Ref.  18:p.  103] 

Program  trends  support  guidance  to  the  point  that  a 
Reliability  Availability  Maintainability  Working  Group 
(RAMWG)  was  organized  subordinate  to  the  Test  Integration 
Working  Group  (TIWG) .  This  group  was  responsible  for 
developing  reliability  requirements  and  developing  Failure 
Definition  and  Scoring  Criteria  (FDSC) .  The  TIWG  was 
comprised  of  representatives  from  the  test  and  evaluation 
community^  the  material  developer  and  the  user.  The  TIWG 
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met  quarterly  initially,  then  monthly  and  bimonthly  during 
tests  to  score  test  incidents . 

An  advantage  of  this  organization  is  that  it  is  an 
effective  way  to  coordinate  testing  for  reliability.  The 
group  is  able  to  easily  crosswalk  reliability  requirements 
to  test  plans  and  scenarios,  ensuring  test  events  are 
scheduled  to  measure  reliability  growth.  This  organization 
also  ensures  FDSC  are  clearly  defined.  This  practice 
results  in  efficient  scoring  conferences  due  to  the  prior 
coordination,  definition  of  failures,  and  familiarity  with 
test  events  and  incidents. 

A  disadvantage  of  this  organization  is  that  the  group 
may  be  focused  more  on  results  and  on  pass/fail  testing 
rather  than  on  influencing  design  and  progressive  growth  to 
meet  requirements.  The  subordinate  relationship  may  also 
diminish  the  value  of  the  reliability  group,  making 
reliability  seem  less  important  than  testing  overall. 

The  importance  of  reliability,  relative  to  the 
individual  primarily  responsible  for  its  achievement,  also 
varied  by  program.  Most  importantly,  every  program  had  at 
least  one  individual  who  was  primarily  responsible  for 
reliability.  The  level  of  responsibility  varied  from 
program  to  program.  Individual  primary  responsibility  was 
assigned  to  a  Reliability  and  Maintainability  Branch  Chief, 
a  Reliability  Engineer,  or  a  Mechanical  Engineer  in  each 
program.  An  advantage  of  assigning  reliability  to  a  branch 
chief  is  that  reliability  gains  greater  importance  and  has 
a  more  powerful  advocate.  It  is  also  advantageous  to  have 
a  trained  and  experienced  Reliability  Engineer  supporting 
the  program.  A  Reliability  Engineer  was  provided  by  matrix 
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support  in  some  programs.  An  advantage  of  matrix  support 
is  that  the  person  is  usually  skilled  because  they  work  on 
more  than  one  program  and  the  overhead  costs  of  maintaining 
that  person  are  reduced  because  they  are  shared  with  the 
other  programs  and  are  not  a  full-time  employee.  A 
disadvantage  of  matrix  support  is  that  the  person  is  not 
dedicated  to  only  one  program  and  may  be  working  on  several 
at  a  time.  This  may  result  in  a  lack  of  complete 
familiarity  with  the  program  that  a  dedicated  program 
professional  may  have. 

2 .  Reliability  Requirement 

AR  70-1  states:  "MATDEVs  are  to  participate  in  the 
combat  or  training  developer  efforts  to  establish  R&M  and 
other  system  requirements."  [Ref.  16:p.  18]  DA  PAM  70-3 
states  requirements  are  determined  using  the  "IPT  process... 
[and]  reflect  what  the  MATDEV  deems  affordable  and 
technically  achievable  with  program  funding,  risk,  and  time 
constraints."  [Ref.  18:p.  99] 

Program  processes  are  in  accordance  with  documented 
guidance.  In  all  programs,  a  program  reliability 
representative  was  able  to  provide  input  for  establishing 
reliability  requirements  and  helped  create  the  RAM 
Rationale  Report.  The  reliability  representatives  analyzed 
and  adjusted  the  user's  need  considering  industrial 
technological  capability. 

Advantages  of  including  the  program  reliability 
representative  in  the  requirements  development  process  is 
that  the  program  representative  is  brought  into  the  system 
lifecycle  at  origination  and  made  to  feel  a  member  of  the 
team.  The  representative  also  better  understands  the 
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requirements  and  the  need  supporting  the  requirements.  The 
program  representative  also  provides  essential  industrial 
capability  information  that  supports  the  achievability  of 
requirements.  A  disadvantage  of  this  process  is  that  it  is 
possible  that  requirements  that  exceed  industrial 
capability  may  be  reduced,  instead  of  remaining  as  an 
objective  to  induce  industry  to  achieve  and  surpass 
expectations,  realizing  their  full  potential. 

In  developing  reliability  requirements,  all  programs 
used  historical  data  from  similar  previous  systems  as  a 
basis  for  system  reliability,  adjusted  for  technological 
improvements.  For  components,  reliability  requirements 
were  developed  using  three  methods.  The  first  method 
leverages  similar  component  reliability  from  other 
programs.  The  second  method  uses  data  from  the  Rome 
Analysis  Center's,  Reliability  Engineer's  Toolkit.  The 
third  method  was  to  uses  manufacturer's  specifications. 

Historical  information  and  reliability  handbook  data 
can  be  an  excellent  source  for  determining  reliability 
requirements.  There  is  no  better  source  of  information 
than  actual  performance  measurement  of  similar  fielded 
systems.  Care  must  be  taken  when  adjusting  for 
technological  differences  because  arguments  can  be  made 
justifying  an  increase  or  decrease  in  reliability.  As 
identified  in  the  modern  tractor  example  in  Chapter  2, 
modern  systems  are  usually  more  complex  in  design,  which 
may  reduce  reliability.  However,  technological  improvement 
of  components  can  also  increase  reliability,  as  the 
reliability  improvements  are  made  to  a  proven  component  or 
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mechanical  systems  are  replaced  by  more  reliable  electronic 
systems . 

Using  a  manufacturer's  specifications  for  reliability 
requirements  is  not  always  an  effective  method  for 
estimating  reliability  in  military  applications. 
Manufacturers  may  make  claims  as  to  the  reliability  of 
their  systems^  but  unless  their  systems  are  proven  through 
independent  testing  or  through  analysis  of  field  data^ 
claims  should  be  considered  unsubstantiated  until  tested. 
Manufacturers  may  also  develop  specifications  from  faulty 
analysis  or  by  using  prediction  methods  for  systems 
unintended  for  military  tactical  use.  This  is  demonstrated 
in  the  table  below^  which  compares  one  system' s  reliability 
predictions  by  manufacturers^  with  MIL-STD-217  predictions 
for  that  same  system.  This  comparison  resulted  in  the 
Government's  insistence  that  MIL-STD-217  not  be  used  as  a 


predictor  because  of  its  inaccuracy  for  field  data. 


Model 

Predicted  Failures  Per 

Million  Hours 

Bell  Communications  Research 

12,502 

MIL-HDBK-217 

715,784 

British  Telecom 

1,258 

CNET  (French) 

16,714 

Nippon  Telephone  and 

Telegraph 

9,  525 

Note:  "MIL-HDBK  217  is  not  intended  to  predict  field 
reliability  and,  in  general,  does  not  do  a  very  good  job  in 
an  absolute  sense.  The  reasons  for  this  are  numerous 
including  different  failure  definitions  for  field  problems 
that  MIL-HDBK-217  does  not  account  for..."  RAC  Technical 

Brief,  April  1990 

Table  6.  Reliability  Prediction  Comparison  [From  Ref.  7:p. 

10-15] 

AR  70-1  states:  ''Contract  R&M  requirements  should 
reflect  operational  R&M  requirements  in  the  ORD  or  reflect 
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technical  values  derived  from  them."  [Ref.  16:p.  100] 
Programs  followed  this  guidance  in  translating  reliability 
requirements,  outlined  in  contract  performance 
specifications,  for  the  contractor.  In  only  two  programs, 
reliability  requirements  varied  from  those  in  the  ORD.  For 
those  two  programs,  ORD  operational  reliability 
requirements  expressed  in  terms  of  Mean  Miles  Between 
Operational  Mission  Failure  (MMBOMF)  were  translated  into 
hardware  reliability  requirements  expressed  in  terms  of 
Mean  Miles  Between  Hardware  Mission  Failure  (MMBHMF) . 
Requirements  translation  was  accomplished  by  multiplying 
the  MMBOMF  requirement  by  1.33,  a  widely  accepted 
acceleration,  to  arrive  at  the  MMBHMF  requirement.  This 
translation  was  done  so  that  the  contractor  would 
understand  the  system  reliability  requirement  independent 
of  operator  or  maintainer  induced  failures,  and  would  be 
responsible  for  system  reliability  at  the  accelerated  rate. 
Using  a  hardware  reliability  requirement  is  acceptable  for 
developmental  testing,  but  during  operational  testing,  the 
operational  reliability  requirement  must  be  used.  If 
hardware  reliability  requirements,  instead  of  operational 
reliability  requirements,  were  used  during  operational 
testing,  the  contractor  could  not  be  held  responsible  for 
operator  or  maintainer  induced  failures  as  a  result  of 
complex  operating,  doctrine  or  maintenance  procedures.  The 
only  caution  is  that  a  sufficient  margin  of  error  must  be 
allowed  in  translating  operational  reliability  requirements 
to  hardware  requirements.  Additionally,  the  contractor 
must  be  aware  of  the  operational  requirement  and  use  Early 
or  Limited  User  Tests  during  developmental  testing  to 
achieve  this  with  the  system  design. 
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To  manage  reliability  and  increase  visibility, 
requirements  were  also  included  in  the  Acquisition  Program 
Baseline  and  Milestone  III  Exit  Criteria.  Including 
requirements  in  the  Milestone  III  Exit  Criteria  ensured  the 
Milestone  Decision  Authority  would  not  approve  the  system 
for  production  or  fielding  without  achieving  requirements. 

3 .  Reliability  Management 

AR  70-1  states:  "The  MATDEV  is  responsible  for 
development  and  implementation  of  an  effective  R&M 
program...."  "This  applies  to  all  developmental  programs  and 
non-developmental  item  (NDI)  programs,  other  than 
commercial  programs...."  [Ref.  16  :p.  18] 

As  a  result  of  specification  reform,  programs  are 
prevented  from  specifying  to  contractors  "how  to"  design, 
manufacture,  or  test  for  reliability.  To  maintain 
visibility  of  reliability  achievement  over  contractors, 
programs  used  a  variety  of  methods. 

Reviews  were  one  method  used  to  maintain  visibility 
over  reliability  achievement.  A  Corrective  Action 
Management  Review  (CAMR)  consisting  of  both  contractor  and 
program  representatives  worked  with  "Tiger  Teams"  of 
contractor  representatives  that  conducted  root  cause 
analysis  of  failures. 

Two  programs  began  development  prior  to  specification 
reform  and,  for  one  of  these  programs,  it  was  stated  as  the 
reason  the  program  succeeded  in  achieving  reliability 
requirements.  Three  programs  had  Reliability  Program  Plans 
prepared  for  the  program  office,  while  two  other  programs 
had  plans  prepared  that  the  contractor  followed.  Four 
programs  stated  that  Reliability  Program  Plans  were 
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unnecessary  because  of  the  limited  developmental  nature  of 
the  program. 

Although  these  programs  were  successful  in  achieving 
their  reliability  requirements,  they  jeopardized  their 
success  by  not  using  a  Reliability  Program  Plan.  Guidance 
specifies  that  implementation  of  an  R&M  plan  is  required 
for  all  programs,  regardless  of  their  nature.  There  have 
been  many  programs  which  were  thought  to  be  non- 
developmental  or  only  requiring  integration  of  proven 
components  that  instead,  proved  to  be  very  difficult  in 
achieving  reliability  thresholds. 

Primary  sources  of  reliability  guidance  according  to 
respondents,  consisted  of  both  canceled  and  current 
publications.  Publications  cited  included  AR  702-3,  Army 
Materiel  Systems  Reliability,  Availability,  and 
Maintainability  (RAM),  and  AR  70-1,  Army  Acquisition 
Policy,  which  superseded  the  currently  canceled  AR  702-3. 
Another  essential  publication  included  TRADOC  Pamphlet  70- 
11,  RAM  Rationale  Handbook,  which  was  also  canceled.  Cited 
publications  governing  reliability  achievement  included  the 
once  mandatory  MIL-STD-781  and  the  discretionary 
publication  that  superseded  it,  MIL-HDBK-781A,  Reliability 
Testing  for  Engineering  Development ,  Qualification ,  and 
Production.  Additional  popular  publications  were  MIL-HDBK- 
189,  Reliability  Growth  Management,  and  MIL-STD-1 62 9 , 
Procedures  for  Performing  a  Failure  Mode,  Effects  and 
Criticality  Analysis ,  which  was  canceled.  According  to 
respondents,  many  programs  still  used  canceled  or 
superseded  Government  guidance  by  transforming  that 
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guidance  into  performance  specifications  or  directives  in 
the  Statement  of  Work  (SOW) . 

The  cancellation  of  these  sources  of  guidance  was 
primarily  due  to  acquisition  and  specification  reform.  A 
program's  inability  to  specify  standards  that  must  be 
complied  with  for  design,  manufacture,  or  test  for 
reliability  can  be  challenging.  With  few  exceptions, 
military  programs  are  not  allowed  to  specify  standards  and 
task  description  numbers  in  contractor' s  statements  of 
work.  Military  standards  and  specifications  that  were  once 
mandatory  were  changed  to  discretionary  handbooks  as  a 

result  of  changes  made  by  Government  specification  reform. 
These  handbooks  all  have  the  same  forward  that  states  they 
"cannot  be  cited  as  a  requirement.  If  it  is,  the 
contractor  does  not  have  to  comply."  These  changes  were 
made  to  allow  industry  to  use  commercial  standards  and 

specifications  instead  of  stringent  Government  standards, 
which  were  considered  as  cost  drivers  and  unnecessary. 

However,  with  the  cancellation  of  many  military 
specifications,  there  is  still  a  void  in  commercial 
specifications  as  some  are  still  awaiting  publication.  A 
commercial  standard  replacement  for  MIL-STD-785B, 
Reliability  Program  for  Systems  and  Equipment  Development 
and  Production,  canceled  August  1998,  is  still  awaiting 
publication  by  IEEE.  This  is  just  one  example  of  this 

problem.  An  Internet  search  for  reliability  standards  and 
specifications,  commercial  or  otherwise,  returns  more  hits 
for  military  standards  than  any  other,  even  though  many 
have  been  canceled.  It  is  apparent  that  commercial 

industry  still  looks  to  the  military  for  providing  this 
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guidance.  Another  problem  with  commercial  standards  and 
specifications  is  that  many  are  applicable  only  for  use  in 
commercial  environments,  not  demanding  military  tactical 
environments.  Military  deployments  can  subject  a  system  to 
all  of  the  world's  temperature  or  humidity  extremes,  not  to 
mention  dust,  vibration  and  shock. 

4 .  Reliability  Achievement 

Reliability  was  designated  as  a  Key  Performance 
Parameter  (KPP)  for  one  program  while  another  stated  it  was 
more  important  than  a  KPP  and  received  general  officer 
visibility.  All  programs  agreed  that  reliability  was  an 
important  performance  parameter. 

Although  it  may  not  be  necessary  to  designate 
reliability  as  a  KPP  to  successfully  achieve  reliability 
requirements,  it  is  necessary  to  recognize  and  convey  to 
the  contractor  the  importance  of  reliability  and  enforce 
its  achievement.  Designating  reliability  as  a  KPP  would 
help  protect  it  from  trade-offs  for  seemingly  more 
important  considerations  such  as  performance 
characteristics  or  decreased  initial  costs.  Only  recently 
have  program  managers  been  encouraged  to  consider  system 
lifecycle  costs  instead  of  initial  cost  that  might  have 
permitted  buy-ins  (inexpensive  initial  costs  with 
extraordinary  operations  and  support  costs) . 

AR  70-1  states  "assessment  of  R&M  in  accordance  with 
the  FDSC  will  be  an  objective  in  every  system  level  test 
(technical,  operational  and  production)."  [Ref.  16:p.  100] 
AR  70-1  further  states:  "A  R&M  IPT  will  be  held  to  review, 
classify  and  charge  test  data  from  system  level  tests 
planned  for  assessment  of  R&M  requirements."  [Ref.  16  :p. 
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100]  Adherence  to  this  guidance  was  one  reason  given  for 
program  success.  Programs  attributed  success  to  early 
establishment  and  updating  of  Failure  Definition  and 
Scoring  Criteria,  well-managed  scoring  conferences  and 
funded  and  continuous  tests.  Failure  Definition  Scoring 
Criteria  (FDSC)  must  be  clearly  defined  with  consideration 
of  all  possible  failure  scenarios.  Failures  must  be 
defined  with  regard  to  criticality  and  what  constitutes  a 
failure . 

AR  70-1  states:  "MATDEVs  are  to  plan  for  and  manage 
system  R&M  development  and  are  encouraged  to  utilize 
reliability  growth  planning  tools."  [Ref.  16:p.  99]  Some 

programs  implemented  this  guidance  by  establishing  a 
database  for  tracking  failures  and  root  cause  analyses. 
Reliability  growth  tools  used  by  programs  included  Failure 
Mode  Effects  and  Criticality  Analysis  (FMECA) ,  Test- 
Analyze-Fix-Test  (TAFT) ,  fishbone  cause  and  effect.  Failure 
Reporting  and  Corrective  Action  System  (FRACAS) ,  and 
component  level  Environmental  Stress  Screening  (ESS) , 
Highly  Accelerated  Life  Testing  (HALT)  and  Highly 
Accelerated  Stress  Testing  (HAST) .  Although  these  are  all 
good  reliability  growth  tools  that  work  to  identify  causes 
for  failure,  these  tools  are  primarily  reactive  to  failures 
instead  of  being  proactive,  and  working  to  prevent 
failures.  One  tool,  FMECA,  does  attempt  to  influence 
design  by  analyzing  potential  system  failures.  No  programs 
indicated  the  use  of  Physics  of  Failure  (PoF) ,  a 
reliability  design  tool  that  uses  Modeling  &  Simulation 
(M&S)  to  "identify  first-order  failure  mechanisms  prior  to 
physical  testing."  [Ref.  4:p.  10-16]  One  program  stated 
that  it  "didn't  work  to  influence  design"  because  it  was 


65 


merely  integrating  non-developmental  items.  Reactive  tools 
may  be  effective  when  used  in  this  situation,  but 
developmental  programs  cannot  wait  until  after  the  design 
is  established  to  measure  reliability  and  could  benefit 
from  use  of  a  predictive  modeling  tool  to  evaluate  trade¬ 
offs  and  assist  in  design. 

Incentivizing  a  contractor  can  be  effective  for 
achieving  and  exceeding  reliability  requirements.  However, 
only  two  programs  incentivized  their  contractor  for 
troublesome  components.  Others  stated  that  there  was  no 
need  to  do  this  because  the  contractor  knew  what  had  to  be 
done  and  continuing  the  program  was  incentive  enough. 

Regarding  reliability  sustainment  after  fielding,  one 
program  hadn't  thought  about  it  and  stated  that  the 
logisticians  were  primarily  concerned  with  that.  Other 
programs  planned  on  using  service  representatives  and 
review  boards  to  identify  and  correct  failures.  Another 
program  would  use  Deficiency  Product  Reports  (DPR)  and 
Contractor  Logistics  Support  (CLS) .  One  last  program 
thought  the  Army  could  do  a  better  job  of  tracking  parts 
and  system  reliability  to  better  improve  faulty  components. 

The  comment  by  one  program  Reliability  Engineer  that 
it  hadn't  considered  reliability  after  fielding  and  it  was 
the  responsibility  of  the  logisticians  is  disconcerting. 
This  comment  is  equivalent  to  a  asking  a  production  line 
worker  "Who  is  responsible  for  Quality  Assurance/Control?" 
and  getting  the  response,  "They  are,  down  there." 
Reliability  at  all  stages  should  be  a  concern  of  everyone 
within  the  program.  Logisticians,  as  well  as  Reliability 
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Engineers,  should  be  concerned  with  it  from  the  beginning 
to  effectively  plan  for  support. 

5 .  Reliability  Timeline 

All  programs  agreed  that  it  is  necessary  to  focus  on 
reliability  early  in  the  lifecycle.  Helping  to  establish 
reliability  requirements  that  are  "agreed  to  and  understood 
by  all"  is  one  of  the  most  important  steps  in  the  process. 
These  reliability  requirements  can  mean  success  or  failure 
for  a  program.  If  requirements  aren't  considered  to  be 
optimal  for  all  stakeholders,  then  there  is  a  chance  that 
"requirements  creep"  can  occur.  This  creep  or  change  of 
requirements  can  cause  a  program  to  spend  additional  time 
or  money  in  attempts  to  meet  increasing  requirements,  or  to 
needlessly  spend  time  and  money  trying  to  achieve  a 
requirement  that  is  later  deemed  unrealistic  and  then 
reduced . 

The  next  important  step  is  to  establish  Failure 
Definition  and  Scoring  Criteria  (FDSC)  that  are  easily 
interpreted  and  understood  by  all.  If  FDSC  do  not  clearly 
outline  what  constitutes  a  failure,  much  time  can  be  spent 
arguing  for  one  interpretation  or  another.  Failure 
definitions  are  prepared  for  all  possible  eventualities  to 
ensure  that  failures  are  recognized  and  properly  scored. 

In  designing  the  system,  it  is  important  to  influence 
the  design  to  increase  reliability  by  using  a  design 
analysis  tool  such  as  PoF.  Poorly  planned  designs  can 
greatly  increase  system  lifecycle  costs.  This  is  because 
once  a  design  is  established,  future  program  costs  are 
often  more  than  70%  established.  [Ref.  4:p.  12-5] 
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Last  is  the  test-analyze-f ix-test  (TAFT)  process. 
This  process  tests  the  component  or  system,  analyzes 
failures,  fixes  the  failures  and  retests  to  ensure  the 
failure  was  corrected  and  other  failures  didn't  occur  as  a 
result  of  the  fix.  Testing  is  an  important  step  and  must 
occur  throughout  the  system's  lifecycle.  Developmental 
testing  is  important  to  reduce  risk  and  confirm 
specifications.  Operational  tests  determine  if  the  system 
is  operationally  effective  and  suitable. 

6 .  Ultra-reliability 

Most  survey  respondents  were  unfamiliar  with  the  term 
ultra-reliability  or  had  no  experience  in  attempts  to 
achieve  it.  Respondents  did  agree  that  designating  ultra¬ 
reliability  as  a  KPP  depended  on  its  achievability  and  on 
the  system,  the  operating  environment,  and  use  of 
commercial  items.  Mechanical  systems  programs  believed 
that  ultra-reliability  goals  were  unachievable  for  their 
systems  due  to  the  demanding  operating  environment. 
Electronic  system  programs  believed  it  was  achievable  if 
sufficient  time  and  money  were  available. 

Ultra-reliability  is  considered  a  means  for  reducing 
the  logistics  burden  and  decreasing  lifecycle  costs,  but 
the  recommendation  for  designating  ultra-reliability  as  a 
KPP  met  resistance  from  some  high  ranking  Army  officials 
who  argued  that  it  was  too  soon  in  the  lifecycle  to  do 
this,  and  it  would  also  reduce  trade  space.  If  ultra¬ 
reliability  is  not  established  as  a  KPP  at  the  start  of  the 
program,  then  later  might  be  too  late  to  try  to  achieve  it 
due  to  the  necessity  to  design  in  ultra-reliability  from 
the  beginning.  "Key  trade-offs  between  [components]...  and 
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design  layout  occur  early  in  a  program  and  can 
significantly  impact  the  reliability  and  lifecycle  costs  of 
a  system."  [Ref.  31] 

Modeling  and  Simulation  tools,  such  as  Physics  of 
Failure,  exist  for  development  of  systems  with  ultra¬ 
reliability  and  some  systems,  such  as  the  Space  Shuttle, 
are  working  to  achieve  it.  It's  true  that  it  requires  a 
substantial  initial  investment,  however,  the  benefits,  such 
as  reduced  O&S  costs  and  lives  saved,  will  make  a  valuable 
but  intangible  return  on  the  initial  investment  many  times 
over . 

C.  SUMMARY 

Analysis  of  the  interview  data  shows  that  there  are 
many  successful  practices  and  strategies  in  use.  Some  of 
these  practices  support  documented  guidance,  while  others 
are  independent  of  guidance. 

Successful  programs  treated  reliability  as  an 
important  goal  and  enjoyed  the  support  of  management.  They 
were  sure  to  focus  on  reliability  from  the  beginning  and 
continued  to  mature  it  through  definition  of  requirements, 
measures,  analysis,  and  testing,  fixing  and  retesting. 

Programs  identified  that  tracking  reliability  for 
fielded  systems  and  components  for  their  own  use  as  well  as 
other  programs  using  similar  components  and  systems  could 
be  improved.  Programs  also  stated  that  the  Government  must 
reinstate  or  publish  more  mandatory  reliability  guidance  to 
better  manage  its  achievement. 

As  for  ultra-reliability,  most  mechanical  programs 
believed  it  wasn't  applicable  to  their  types  of  systems  and 

wouldn't  be  achievable  without  an  extraordinary  investment 
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of  time  and  money.  Electronic  programs  thought  it  could  be 


achieved, 

components 


but  should  only  be  considered  for  safety  critical 
or  systems . 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  INTRODUCTION 

Because  only  20%  of  Army  weapon  system  programs  meet 
reliability  requirements^  it  is  essential  to  capture 
successful  strategies  and  practices.  Analysis  of  the 
subjective  questionnaire  data  gathered  and  presented  in  the 
previous  chapter  reveals  many  interesting  practices  used  by 
the  few  successful  programs  to  achieve  reliability 
requirements.  in  this  chapter^  conclusions  are  drawn 
regarding  successful  practices  and  recommendations  are  made 
for  implementation  of  these  practices  for  programs 
attempting  to  achieve  reliability  requirements. 

B.  RESTATEMENT  OF  RESEARCH  QUESTIONS 

Preliminary  research  questions  are  addressed  here  with 
a  short  summary  of  the  answer. 

•  What  is  reliability  and  how  does  it  affect  total 
life  cycle  costs  and  the  logistics  burden? 

Reliability  is  ''the  probability  that  an  item  will 
perform  its  intended  function  for  a  specified  interval 
under  stated  conditions."  [Ref.  2:p.  36]  Generally^ 

reliability  is  most  often  modeled  as  an  exponential 
distribution  that  is  a  function  of  MTBF  and  time^  and  as 
time  increases^  reliability  decreases. 

An  investment  of  one  dollar  in  reliability  was  proven 
to  decrease  lifecycle  costs  by  eight  dollars  in  one  case 
and  five  dollars  in  another  case.  With  this  savings^  funds 
may  be  redirected  to  more  pressing  needs  such  as 
modernizing  the  force  or  quality  of  life  improvement. 
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Reliability  has  the  effect  of  reducing  the  logistics 
burden.  More  reliable  systems  require  less  spares  and 
maintenance,  reducing  the  logistics  footprint  on  the 
battlefield . 

•  What  is  ultra-reliability  and  why  is  this  a 
recommended  goal  of  FCS? 

Ultra-reliability  is  recognized  as  an  extremely  high 
level  of  reliability,  however,  there  is  no  standard 
numerical  reliability  figure  or  definition  associated  with 
it.  The  National  Aeronautic  Space  Administration  (NASA) 
refers  to  ultra-reliability  in  software  as  having  a  failure 
probability  during  a  one-hour  test  mission  of  less  than 
0.0000001  .  This  means  that  the  software  will  not  fail 
99.99999%  of  the  time.  [Ref.  4] 

The  Army  Science  Board  recommended  that  ultra¬ 
reliability  be  a  goal  of  the  FCS  because  it  will  reduce  the 
logistics  burden  by  decreasing  the  number  of  maintenance 
personnel,  equipment,  and  spares  required  for  support. 
Some  estimates  indicate  that  ultra-reliability  can  reduce 
service  and  support  personnel  requirements  in  the  Objective 
Force  Area  of  Operations  by  as  much  as  83%.  [Ref.  12] 

•  What  is  the  guidance  for  achieving  reliability 
requirements? 

Both  mandatory  and  discretionary  guidance  exists  for 
the  achievement  of  reliability  requirements,  but,  because 
of  acquisition  reform,  there  is  not  as  much  mandatory 
guidance  as  there  once  was . 

Limited  mandatory  guidance  for  achieving  reliability 
requirements  for  all  services  is  provided  in  DoDD  5000.1 
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and  DoD  5000. 2-R.  Army  specific  mandatory  guidance  is 
found  in  both  AR  70-1  and  AR  71-9. 

There  are  many  discretionary  sources  of  reliability 
guidance  in  addition  to  the  mandatory  guidance. 
Discretionary  sources  consist  mostly  of  Department  of  the 
Army  Pamphlets  and  Military  Handbooks.  Some  of  the  most 
important  and  management  relevant  guidance  is  found  in  DA- 
PAM  70-3  which  devotes  an  entire  section  to  Reliability, 
Maintainability  and  Availability.  There  are  also  several 
military  handbooks  addressing  reliability  including  MIL- 
HDBK-781A  and  MIL-HDBK-189 . 

There  is  not  as  much  regulatory  guidance  as  there  had 
been  in  the  past.  Many  mandatory  standards  and 

specifications  have  been  canceled  or  made  discretionary  as 
a  result  of  acquisition  reform  in  the  interest  of 
eliminating  cost  drivers  and  giving  preference  to 
commercial  standards  over  military  standards. 

C.  CONCLUSIONS  AND  RECOMMENDATIONS 

The  primary  research  question  of  this  thesis  is: 

What  strategies  should  be  used  to  achieve  reliability 
requirements  for  weapon  system  development? 

1 .  Strategies  of  Successful  Programs 

Secondary  research  questions  supporting  this  question 
focused  on: 

How  have  successful  weapon  system  programs  achieved 
reliability  requirements? 

To  answer  this  question,  nine  successful  programs  were 
questioned  regarding  their  practices . 

a.  Organizing  for  Reliability 


73 


How  did  programs  organize  to  achieve  reliability 
requirements? 

Conclusion :  Most  programs  organized  Reliability 

Working  Groups  subordinately  to  the  Test  Integrated  Working 
Group  (TIWG) .  A  person  primarily  designated  for 

reliability,  usually  a  Reliability  Engineer,  represented 
the  program  in  the  groups . 

Recommendation :  Programs  should  organize  a 

Reliability  Integrated  Process  Team  to  identify,  solve,  and 
manage  reliability  issues.  Instead  of  working 

subordinately  for  the  TIWG,  this  group  should  have 
equivalent  authority,  while  also  working  with  the  TIWG  to 
coordinate  testing. 

b.  Requirements  Development 

How  were  reliability  requirements  developed? 

Conclusion :  Reliability  requirements  were 

developed  with  input  from  both  the  user  and  program 
reliability  representative  based  upon  need,  industrial 
capability,  technological  advancements,  and  historical 
information.  The  collection  of  historical  information  for 
use  by  programs  should  be  vastly  improved.  When  defining 
reliability  requirements,  only  one  program  designated  it  as 
a  Key  Performance  Parameter  (KPP) ,  but  all  other  programs 
treated  reliability  as  an  important  objective. 

Recommendation :  Programs  should  develop  a 

Reliability,  Availability  &  Maintainability  (RAM)  Rationale 
Report  using  an  Integrated  Product  Team  (IPT)  process  based 
upon  historical  information  for  similar  systems  and 
components  with  consideration  given  to  industrial 

capability  and  technological  improvements.  To  prevent  the 
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possibility  of  requirements  being  reduced  due  to  a  lack  of 
technological  capability^  the  requirement  should  remain  as 
the  objective  with  the  technological  capability  as  the 
threshold.  The  contractor  could  then  be  incentivized  to 
achieve  the  objective  requirement. 

The  Standard  Army  Maintenance  System  and 
maintenance  practices  should  be  modified  to  enable  the 
collection  of  reliability  data  for  components  as  well  as 
systems.  Pertinent  necessary  information^  at  a  minimum^ 
that  should  be  collected  includes  hours  or  miles  operated. 
For  depot  level  reparables^  an  analysis  of  the  reason  for 
failure^  performed  by  the  repair  technician^  should  also 
complement  the  operating  time.  The  collected  information 
could  then  be  provided  to  the  program  office  to  design 
decisions  impacting  reliability  issues. 

Programs  should  also  designate  reliability  as  a 
KPP  for  programs  that  are  electronic  in  nature  and  where 
safety  or  system  criticality  is  a  primary  concern.  If 
reliability  is  given  the  same  weight  as  performance^  cost^ 
and  other  factors  during  the  trade-off  process^  it  should 
not  be  unnecessarily  sacrificed. 

c.  Reliability  Management  Plans 

What  management  plans  for  achieving  reliability 
requirements  were  developed  and  implemented? 

Conclusion :  Most  programs  did  not  have 
Reliability  Program  Plans  because  of  the  limited 
developmental  nature  of  their  systems.  Programs  managed 
reliability  achievement  through  RIWGs  that  analyzed  and 
scored  test  data.  Teams  of  contractor  and  program 
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representatives  conducted  root  cause  analysis  to  identify 
and  correct  failures. 


Recommendation :  Programs,  especially 
developmental  ones,  should  develop  and  implement 
Reliability  Program  Plans  to  effectively  manage  reliability 
achievement.  Key  aspects  of  the  Reliability  Program  Plan 
should  be:  organizational  responsibilities;  requirements 
development;  Failure  Definition  and  Scoring  Criteria;  use 
of  design  tools  such  as  Physics  of  Failure;  test  plans; 
root  cause  analysis  methods;  and  incentivizing  for 
achievement . 

d.  Reliability  Achievement 

What  processes  were  used  to  achieve  reliability 
requirements  and  how  was  this  measured? 

Conclusion :  Primarily,  a  test-analyze-f ix-test 
process,  which  encourages  reliability  growth,  was  used  to 
achieve  reliability  requirements.  Reliability  requirements 
were  measured  and  analyzed  during  developmental  testing. 
Many  programs  didn't  attempt  to  design  in  reliability  prior 
to  testing  because  they  were  not  developmental  programs. 
Developmental  programs  would  benefit  from  a  design  analysis 
tool  such  as  Physics  of  Failure  to  develop  the  optimal 
design  prior  to  commitment  of  funds  for  an  unproven  design. 
Non-Developmental  Item  (NDI)  programs  with  significant 
integration  challenges  could  benefit  as  well. 

Contractors  were  incentivized  to  achieve 
reliability  requirements  for  troublesome  components.  It  is 
not  necessary  to  incentivize  a  contractor  for  the  entire 
system,  as  this  could  be  very  costly.  Nor  should  a 
contractor  be  incentivized  from  the  start,  as  it  would  be 
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difficult  to  determine  what  the  most  troublesome  components 
would  be  and  to  determine  how  much  of  an  incentive  should 
be  used. 

Reliability  Engineers  must  be  concerned  with 
reliability  throughout  the  system's  life  instead  of  just 
focusing  on  ''getting  through  testing."  Reliability 
Engineers  must  work  with  and  assist  logisticians^  even 
after  the  system  is  fielded.  This  should  be  a  result  of 
the  Reliability  IPT  process^  instead  of  compartmentalized 
planning  and  development. 

Recommendation :  Programs  should  use  many 
iterations  of  developmental  testing  to  measure  and  "grow" 
reliability  using  the  test-analyze-f ix-test  process. 
Programs  should  also  use  Physics  of  Failure  to  design  in 
reliability  prior  to  testing  to  reduce  the  number  of  costly 
iterations.  Incentives  should  be  focused  on  troublesome 
components^  rewarding  the  contractor  when  requirements  are 
exceeded.  Reliability  Engineers  should  work  closely  with 
logisticians  as  part  of  an  IPT^  to  better  coordinate 
logistics  planning  and  influence  reliability  after 
fielding . 

e.  Reliability  Timeline 

Where  in  the  Acquisition  Process  was  the  program 
focused  on  achievement  of  reliability  requirements? 

Conclusion :  All  programs  agreed  that  reliability 
must  be  a  key  focus  from  the  start  of  the  program.  It's 
important  to  first  define  reliability  requirements  that  are 
acceptable  to  both  the  user  and  program^  define  Failure 
Definition  and  Scoring  Criteria  (FDSC)  to  allocate  failures 
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and  score  them,  and  to  use  an  iterative  test-analyze-f ix- 
test  process  to  achieve  reliability  growth. 

Recommendation :  Programs  must  focus  on 
reliability  early  in  the  lifecycle  before  final  development 
of  the  Operational  Requirements  Document  (ORD) . 

2 .  Ultra-reliability 

Should  ultra-reliability  be  a  goal  of  weapon  system 
programs? 

Conclusion :  Most  programs  agreed  that  ultra¬ 
reliability  is  a  difficult  objective  to  achieve  because  of 
the  immense  investment  of  time  and  money. 

Recommendation :  Ultra-reliability  should  only  be 
implemented  in  those  programs  where  a  cost-benefit  analysis 
demonstrates  that  the  benefits  of  achieving  this  difficult 
objective  outweigh  the  costs.  Consideration  must  be  given 
to  the  technological  risk  and  should  only  be  pursued  for 
electronics  systems  or  critical  safety  related  items . 

3 .  Application  of  Successful  Strategies 

What  are  the  best  strategies  that  should  be  used  for 
the  Future  Combat  Systems  (FCS)  Program  as  well  as  other 
weapon  system  programs? 


Conclusion 

:  The  FCS 

is 

a  program  that 

could 

be 

compared  to  a 

combination 

of 

the  previously 

discussed 

programs.  It 

has  elements 

of 

each  of  these  programs 

in 

that  it  is  a  program,  in  the  developmental  stage,  relying 
heavily  on  mechanical,  electrical,  and  software  systems. 

Recommendation:  The  FCS  program,  as  well  as  other 
programs,  should  implement  the  previous  recommendations  to 
achieve  reliability  requirements. 
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D.  LESSONS  LEARNED/RECOMMENDATIONS  FOR  IMPROVEMENT 

Much  can  be  learned  from  successful  programs.  The 
Acquisition  Logistics  Guide  outlines  many  successful 
reliability  practices  used  by  organizations  in  its 
Reliability  and  Maintainability  chapter.  interviewed 
programs  also  made  good  recommendations  for  changing  steps 
and  processes  concerning  reliability  achievement. 

AR  70-1  states:  ''MATDEVs  are  to  track  fielded  systems 
failure  and  repair  histories  starting  at  First  Unit 
Equipped  (FUE)."  [Ref.  16:p.  99]  One  program  stated  that 
this  could  be  improved  upon.  A  recommendation  made  for 
process  improvement  was  for  the  Army  maintenance  system  to 
be  held  responsible  for  collecting  reliability  failure  data 
for  components  in  service.  The  benefits  of  this 
recommendation  are  that  the  programs  responsible  for  these 
systems  and  components  would  gain  better  visibility  of 
failures^  their  causes^  and  recommendations  for 
improvement.  Other  programs  could  also  benefit  by  having 
field  reliability  data  that  could  be  used  for  comparison^ 
estimation  of  future  requirements^  and  even  selection  of 
contractors  and  components.  Currently^  programs  only  have 
visibility  of  parts  requirements  or  demand  trends  without 
knowing  the  length  of  time  the  part  operated^  operating 
conditions^  and  reasons  for  failure.  LARs  at  the 
installation  level  may  sometimes  investigate  and  report 
failure  trends  identified  in  the  field.  if  programs  are 
notified  or  aware  of  failures^  they  may  also  investigate 
causes  and  if  correctable^  initiate  engineering  change 
proposals.  Reliability  Centered  Maintenance  programs  have 
also  been  used  to  develop  a  scheduled  maintenance  program 
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to  identify  and  prevent  failures  by  replacing  parts  before 
it  affects  system  readiness. 

Programs  also  stated  that  there  was  not  enough 
guidance  concerning  reliability  management  and  achievement. 
They  thought  that  the  Government  had  gone  too  far  in 
canceling  many  military  specifications  and  standards  and 
that  some  should  be  reinstated. 

Another  idea  by  one  program  for  a  process  change  would 
be  to  allow  ''participation  of  the  system  contractor"  in 
operational  testing.  Law  prohibits  contractors  from 
participating  in  operational  testing  to  prevent  them  from 
taking  measures  to  ensure  the  system  passes.  The  reason 
for  this  is  to  ensure  they  don't  try  to  step  in  to  correct 
failures  or  other  problems  prior  to  completion  of  the  test^ 
altering  the  test  conditions^  and  preventing  a  true  measure 
of  the  system's  performance.  However^  one  program  found 
that  contractors  could  observe  problems  during  testing  as 
they  occurred  without  making  corrections^  so  that  they 
better  understood  the  problems  and  failures  and  their 
ramifications.  By  doing  this^  contractors  knew  what 
changes  to  make  to  allow  the  system  to  successfully 
complete  testing  at  a  later  date. 

E.  RECOMMENDATIONS  FOR  FURTHER  STUDY 

The  scope  of  this  thesis  dealt  only  with  successful 
Army  programs  that  had  recently  and  successfully  completed 
testing  with  respect  to  reliability.  This  researcher  did 
not  consider  other  services  or  civilian  organizations 
because  the  researcher  was  primarily  interested  in  Army 
programs  and  the  environment  in  which  they  operated. 

Recommendations  for  further  study  include: 
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1.  A  comparison  of  the  practices  of  the  Army  with 
other  services  and  civilian  organizations  in  achieving 
reliability  requirements . 

2.  A  study  of  the  methods  that  the  National 
Aeronautic  Space  Administration  uses  to  achieve  reliability 
requirements^  focusing  especially  on  ultra-reliability. 

3.  The  perspective  of  the  civilian  contractor 
counterpart  in  achieving  reliability  requirements. 

4.  A  study  comparing  reliability  predictions^ 
measurements  from  testing^  and  actual  field  data  to 
determine  the  effectiveness  of  the  predictions  and  testing. 

F.  THESIS  SUMMARY 

Reducing  the  logistics  burden  is  a  current  focus  for 
the  Army  as  it  works  to  develop  and  field  the  Objective 
Force.  Increasing  reliability  is  a  key  method  of  achieving 
this  goal^  with  an  added  benefit  of  reducing  O&S  costs. 

Many  programs  are  struggling  to  achieve  even  half  of 
their  established  reliability  requirement.  Therefore^  we 
should  look  to  successful  programs  to  show  us  the  way  in 
achieving  requirements.  If  we  are  unsuccessful  in  our 
endeavors^  future  forces  will  be  unnecessarily  burdened  by 
our  mistakes  and  incapable  of  progress. 

The  Army  aviation  community  is  currently  experiencing 
the  effects  of  attempting  to  manage  this  problem  by 
maintaining  an  aging  fleet.  COL  Stephen  Mundt^  aviation 
director  in  the  office  of  the  Army's  deputy  chief  of  staff 
for  programs  concluded  ''maintenance  and  readiness  issues  in 
our  aviation  fleet  render  our  current  aviation  force 
unaffordable."  [Ref.  32]  Further^  the  aviation  branch  is 
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questioning  its  ability  to  support  the  future  Objective 
Force  because  of  modernization  difficulties  as  a  result  of 
budget  limitations. 

If  we  are  to  transition  to  and  field  an  effective 
Objective  Force  and  other  future  forces,  we  must  be 
successful  in  improving  reliability  in  systems  under 
development.  Otherwise,  our  mistakes  will  degrade  our 
effectiveness  and  decrease  our  capabilities  for  years  to 
come . 
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APPENDIX  A.  INTERVIEW  SUBJECT  QUESTIONNAIRE 


This  appendix  contains  a  copy  of  the  interview 
questionnaire  that  was  administered  to  the  study  subjects. 


Successful  Strategies  for  Achieving  Reliability 
Requirements  Questionnaire 


Name : 

Job  Title : 

Organization : 

Life  Cycle  Phase  of  Program: 

Program  ACAT  Level : 

Date  of  OT : 

1.  Organizing  for  Reliability 

a.  How  was  your  program  organized  for  reliability  (e.g., 
was  there  a  reliability  IPT,  who  was  represented,  what  was 
their  purpose,  and  how  often  did  you  meet) ? 

b.  Who  within  the  program  was  primarily  responsible  for 
reliability  (e.g.,  was  there  one  person  or  more  and  what 
was  their  job  title  and  job  description)? 

2 .  Requirement  Development 

a.  How  much,  if  any  input  did  you  have  in  establishing 
reliability  requirements? 

b.  How  were  reliability  requirements  determined  (e.g., 
were  they  based  on  historical  information,  derived  goals, 
or  another  method) ? 

c.  What  was  your  requirement  for  reliability,  how  was  it 
worded  by  both  the  user  and  to  the  contractor,  and  in  what 
documents  was  it  specified  to  the  contractor? 

3 .  Reliability  Management 

a.  Because  AR  70-1,  Army  Acquisition  Policy,  directs  that 
solicitations  should  not  specify  'how  to'  design, 
manufacture  or  test  for  reliability,  how  did  the  contractor 
manage  reliability  growth  and  achievement  and  what 
oversight  did  you  have  over  the  contractor's  methods. 
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b.  Did  you  have  a  Reliability  Program  Plan,  and  if  so,  did 
you  use  a  documented  template,  and  what  were  its  key 
elements? 

c.  How  much  of  the  plan  did  you  follow?  Did  you  vary  your 
approach? 

d.  What  were  your  primary  sources  of  reliability  guidance 
and  do  you  feel  that  there  is  adequate  amount  of  guidance 
regarding  reliability,  why  or  why  not? 

4 .  Reliability  Achievement 

a.  Was  your  reliability  requirement  a  Key  Performance 
Parameter  (KPP)  and  if  not,  how  much  emphasis  did  its 
achievement  receive? 

b.  What  reliability  did  you  actually  achieve? 

c.  How  would  you  rate  your  program's  achievement  of 
reliability  requirements  (extremely  successful,  moderately 
successful,  or  average)  and  why  do  you  rate  yourself  at 
that  level? 

d.  To  what  do  you  attribute  your  success,  and  did  you  do 
something  that  you  feel  is  innovative  or  extraordinary? 

e.  What  design  analysis  tools  or  process  did  you  primarily 
use,  and  why,  for  failure  root  cause  analysis? 

f.  What  role  did  testing  play  in  achieving  reliability 
requirements  and  how  much  and  what  types  were  conducted? 

g.  When  comparing  the  reliability  figure  measured  in 
Operational  Testing  to  that  from  Developmental  Testing  or 
estimates,  roughly  what  percent  of  an  increase  or  decrease 
was  experienced  and  to  what  do  you  attribute  the  change? 

h.  What  were  the  key  sources  of  unreliability  (i.e.  was  it 
hardware  or  software,  what  was  its  function,  and  why  was  it 
so  troublesome) ? 

i.  Did  you  incentivize  the  contractor  in  any  way  to 
achieve  reliability  requirements,  why  or  why  not? 
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j .  How  do  you  plan  to  continue  to  maintain  the  system' s 
reliability  over  time  after  fielding? 

5 .  Reliability  Timeline 

a.  When  did  you  begin  focusing  on  reliability  (i.e.,  at 
what  phase  or  step  in  the  acquisition  life  cycle)  and  why 
was  it  important  to  be  involved  at  that  time) ? 

b.  What  are  some  of  the  most  important  steps  by  phase 
affecting  reliability  in  the  acquisition  life  cycle  that 
must  be  accomplished? 

6 .  Lessons  Learned/Recommendations  for  Improvement 

a.  If  you  had  the  power  to  change  any  step,  requirement, 
or  process  for  reliability  achievement,  what  would  you 
change  and  why? 

b.  What  are  some  suggestions  that  you  would  like  to  pass 
to  any  program  attempting  to  achieve  requirements?  (Please 
provide  at  least  three) 

c.  What  are  some  of  the  common  pitfalls  to  be  avoided? 

7 .  Ultra-Reliability 

a.  Should  ultra-reliability  be  a  KPP  for  weapon  system 
programs,  why  or  why  not? 

b.  Is  ultra-reliability  achievable  in  the  current 
environment  and  is  it  worthwhile,  why  or  why  not? 
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APPENDIX  B.  LETTER  OF  INTRODUCTION 


Dear  Sir: 

Your  program  was  identified  by  ATEC  as  an  exemplary 
organization  for  the  achievement  of  reliability 
requirements  as  measured  by  Operational  Testing. 

I  am  completing  my  Master  degree  thesis  at  the  Naval 
Postgraduate  School  in  Monterey,  CA.  The  title  of  my 
thesis  is,  "Successful  Strategies  for  Achieving  Reliability 
Requirements  in  Weapon  Systems  Acquisition."  I  expect  to 
graduate  in  March,  so  time  is  of  the  essence  in  gathering 
my  data. 

Therefore,  I  am  conducting  a  survey  regarding  your 
successful  achievement  of  reliability  requirements  to 
explore  your  command/agency' s  strategies  and  methodology. 
Because  you  are  one  of  only  five  exemplary  programs,  it  is 
essential  that  you  are  as  detailed  as  possible  when 
answering  the  questions  to  fully  capture  your  strategies. 

My  seven  areas  of  interest  concern:  1)  organization; 

2)  requirements  development;  3)  reliability  management;  4) 
measures  of  achievement;  5)  where  in  the  acquisition 
lifecycle  you  focused  your  efforts;  6)  lessons  learned;  and 
7)  your  thoughts  on  ultra-reliability.  I  believe  that  many 
programs  could  benefit  from  your  practices,  as  many  are 
struggling  with  achievement  of  reliability  requirements. 

Please  respond  to  the  attached  questionnaire  by 
opening  it,  filling  in  responses  after  each  question,  and 
saving  it,  then  attaching  it  to  an  email  and  return  it  by 
Friday,  February  22.  If  you  have  any  questions  regarding 
the  questionnaire,  please  email  me.  If  you  would  prefer  to 
conduct  a  telephonic  interview  using  the  provided 
questions,  I  would  be  happy  to  give  you  a  call  in  the  next 
week  at  your  convenience.  Please  let  me  know  when  I  should 
call  by  emailing  me  at  jmthorne@nps . navy . mil 
<mailto : jmthorne@nps . navy . mil>,  or  leave  a  voice  message  at 
(831)  899-8614. 

I  will  summarize  all  input  responses  in  the  analysis 
of  data.  If  you  desire,  I  can  forward  a  finished  copy  of 
my  findings. 
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I  thank  you  for  your  assistance  and  with  the  immediate 
response . 


questionnaires. dot 

Sincerely, 

CPT  Jim  Thorne 

Army  Acquisition  Corps 
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